2023-04-20 20:53:34,064 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9 2023-04-20 20:53:34,077 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-20 20:53:34,094 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-20 20:53:34,094 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/cluster_952dfd84-3d7a-970e-16de-5e4c99b46c34, deleteOnExit=true 2023-04-20 20:53:34,094 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-20 20:53:34,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/test.cache.data in system properties and HBase conf 2023-04-20 20:53:34,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/hadoop.tmp.dir in system properties and HBase conf 2023-04-20 20:53:34,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/hadoop.log.dir in system properties and HBase conf 2023-04-20 20:53:34,097 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-20 20:53:34,097 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-20 20:53:34,098 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-20 20:53:34,213 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-20 20:53:34,655 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-20 20:53:34,661 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-20 20:53:34,661 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-20 20:53:34,662 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-20 20:53:34,662 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-20 20:53:34,663 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-20 20:53:34,663 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-20 20:53:34,664 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-20 20:53:34,664 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-20 20:53:34,664 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-20 20:53:34,665 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/nfs.dump.dir in system properties and HBase conf 2023-04-20 20:53:34,665 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/java.io.tmpdir in system properties and HBase conf 2023-04-20 20:53:34,666 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-20 20:53:34,666 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-20 20:53:34,667 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-20 20:53:35,317 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-20 20:53:35,321 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-20 20:53:35,636 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-20 20:53:35,839 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-20 20:53:35,860 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-20 20:53:35,898 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-20 20:53:35,931 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/java.io.tmpdir/Jetty_localhost_44005_hdfs____.pc2un5/webapp 2023-04-20 20:53:36,080 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44005 2023-04-20 20:53:36,120 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-20 20:53:36,120 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-20 20:53:36,550 WARN [Listener at localhost/34685] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-20 20:53:36,712 WARN [Listener at localhost/34685] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-20 20:53:36,732 WARN [Listener at localhost/34685] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-20 20:53:36,738 INFO [Listener at localhost/34685] log.Slf4jLog(67): jetty-6.1.26 2023-04-20 20:53:36,744 INFO [Listener at localhost/34685] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/java.io.tmpdir/Jetty_localhost_38851_datanode____4zb4xi/webapp 2023-04-20 20:53:36,852 INFO [Listener at localhost/34685] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38851 2023-04-20 20:53:37,246 WARN [Listener at localhost/45619] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-20 20:53:37,808 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd3dc40718fafff16: Processing first storage report for DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34 from datanode bab51f44-7c0d-4a22-baba-7df7faaf57e6 2023-04-20 20:53:37,810 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd3dc40718fafff16: from storage DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34 node DatanodeRegistration(127.0.0.1:42965, datanodeUuid=bab51f44-7c0d-4a22-baba-7df7faaf57e6, infoPort=39419, infoSecurePort=0, ipcPort=45619, storageInfo=lv=-57;cid=testClusterID;nsid=152792999;c=1682024015396), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-20 20:53:37,810 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd3dc40718fafff16: Processing first storage report for DS-0541924c-6de1-4246-80d5-1d5dc72a5dff from datanode bab51f44-7c0d-4a22-baba-7df7faaf57e6 2023-04-20 20:53:37,811 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd3dc40718fafff16: from storage DS-0541924c-6de1-4246-80d5-1d5dc72a5dff node DatanodeRegistration(127.0.0.1:42965, datanodeUuid=bab51f44-7c0d-4a22-baba-7df7faaf57e6, infoPort=39419, infoSecurePort=0, ipcPort=45619, storageInfo=lv=-57;cid=testClusterID;nsid=152792999;c=1682024015396), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-20 20:53:37,870 DEBUG [Listener at localhost/45619] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9 2023-04-20 20:53:37,945 INFO [Listener at localhost/45619] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/cluster_952dfd84-3d7a-970e-16de-5e4c99b46c34/zookeeper_0, clientPort=62055, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/cluster_952dfd84-3d7a-970e-16de-5e4c99b46c34/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/cluster_952dfd84-3d7a-970e-16de-5e4c99b46c34/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-20 20:53:37,961 INFO [Listener at localhost/45619] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62055 2023-04-20 20:53:37,970 INFO [Listener at localhost/45619] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 20:53:37,972 INFO [Listener at localhost/45619] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 20:53:38,637 INFO [Listener at localhost/45619] util.FSUtils(471): Created version file at hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9 with version=8 2023-04-20 20:53:38,637 INFO [Listener at localhost/45619] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/hbase-staging 2023-04-20 20:53:38,940 INFO [Listener at localhost/45619] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-20 20:53:39,412 INFO [Listener at localhost/45619] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-20 20:53:39,445 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 20:53:39,446 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-20 20:53:39,446 INFO [Listener at localhost/45619] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-20 20:53:39,446 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 20:53:39,446 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-20 20:53:39,606 INFO [Listener at localhost/45619] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-20 20:53:39,690 DEBUG [Listener at localhost/45619] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-20 20:53:39,802 INFO [Listener at localhost/45619] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44655 2023-04-20 20:53:39,812 INFO [Listener at localhost/45619] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 20:53:39,815 INFO [Listener at localhost/45619] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 20:53:39,835 INFO [Listener at localhost/45619] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44655 connecting to ZooKeeper ensemble=127.0.0.1:62055 2023-04-20 20:53:39,875 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:446550x0, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-20 20:53:39,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44655-0x101a35999ca0000 connected 2023-04-20 20:53:39,905 DEBUG [Listener at localhost/45619] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-20 20:53:39,906 DEBUG [Listener at localhost/45619] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 20:53:39,910 DEBUG [Listener at localhost/45619] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-20 20:53:39,917 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44655 2023-04-20 20:53:39,918 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44655 2023-04-20 20:53:39,919 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44655 2023-04-20 20:53:39,919 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44655 2023-04-20 20:53:39,920 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44655 2023-04-20 20:53:39,927 INFO [Listener at localhost/45619] master.HMaster(444): hbase.rootdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9, hbase.cluster.distributed=false 2023-04-20 20:53:40,003 INFO [Listener at localhost/45619] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-20 20:53:40,004 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 20:53:40,004 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-20 20:53:40,004 INFO [Listener at localhost/45619] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-20 20:53:40,004 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 20:53:40,005 INFO [Listener at localhost/45619] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-20 20:53:40,012 INFO [Listener at localhost/45619] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-20 20:53:40,016 INFO [Listener at localhost/45619] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39223 2023-04-20 20:53:40,019 INFO [Listener at localhost/45619] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-20 20:53:40,027 DEBUG [Listener at localhost/45619] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-20 20:53:40,028 INFO [Listener at localhost/45619] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 20:53:40,032 INFO [Listener at localhost/45619] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 20:53:40,035 INFO [Listener at localhost/45619] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39223 connecting to ZooKeeper ensemble=127.0.0.1:62055 2023-04-20 20:53:40,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:392230x0, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-20 20:53:40,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39223-0x101a35999ca0001 connected 2023-04-20 20:53:40,040 DEBUG [Listener at localhost/45619] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-20 20:53:40,042 DEBUG [Listener at localhost/45619] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 20:53:40,044 DEBUG [Listener at localhost/45619] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-20 20:53:40,045 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39223 2023-04-20 20:53:40,045 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39223 2023-04-20 20:53:40,046 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39223 2023-04-20 20:53:40,047 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39223 2023-04-20 20:53:40,049 DEBUG [Listener at localhost/45619] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39223 2023-04-20 20:53:40,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:40,063 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-20 20:53:40,064 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:40,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-20 20:53:40,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-20 20:53:40,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:40,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-20 20:53:40,087 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,44655,1682024018782 from backup master directory 2023-04-20 20:53:40,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-20 20:53:40,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:40,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-20 20:53:40,091 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-20 20:53:40,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:40,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-20 20:53:40,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-20 20:53:40,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/hbase.id with ID: a989deef-709a-42f8-a284-90450b819dbd 2023-04-20 20:53:40,245 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 20:53:40,260 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:40,705 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7fd3a1b4 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:40,742 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4762129, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:40,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:53:40,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-20 20:53:40,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-20 20:53:40,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-20 20:53:40,790 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:53:40,795 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:53:40,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-20 20:53:40,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store-tmp 2023-04-20 20:53:40,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:40,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-20 20:53:40,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:53:40,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:53:40,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-20 20:53:40,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:53:40,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:53:40,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-20 20:53:40,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/WALs/jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:40,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44655%2C1682024018782, suffix=, logDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/WALs/jenkins-hbase4.apache.org,44655,1682024018782, archiveDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/oldWALs, maxLogs=10 2023-04-20 20:53:40,945 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42965,DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34,DISK] 2023-04-20 20:53:40,954 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:53:41,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/WALs/jenkins-hbase4.apache.org,44655,1682024018782/jenkins-hbase4.apache.org%2C44655%2C1682024018782.1682024020897 2023-04-20 20:53:41,023 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42965,DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34,DISK]] 2023-04-20 20:53:41,024 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:53:41,024 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:41,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-20 20:53:41,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-20 20:53:41,095 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-20 20:53:41,103 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-20 20:53:41,131 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-20 20:53:41,148 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:41,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-20 20:53:41,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-20 20:53:41,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-20 20:53:41,183 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:53:41,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60823937, jitterRate=-0.09365271031856537}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-20 20:53:41,184 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-20 20:53:41,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-20 20:53:41,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-20 20:53:41,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-20 20:53:41,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-20 20:53:41,214 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-20 20:53:41,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 34 msec 2023-04-20 20:53:41,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-20 20:53:41,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-20 20:53:41,282 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-20 20:53:41,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-20 20:53:41,316 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-20 20:53:41,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-20 20:53:41,325 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-20 20:53:41,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-20 20:53:41,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:41,335 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-20 20:53:41,335 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-20 20:53:41,349 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-20 20:53:41,355 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-20 20:53:41,355 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-20 20:53:41,355 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:41,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,44655,1682024018782, sessionid=0x101a35999ca0000, setting cluster-up flag (Was=false) 2023-04-20 20:53:41,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:41,378 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-20 20:53:41,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:41,386 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:41,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-20 20:53:41,393 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:41,396 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.hbase-snapshot/.tmp 2023-04-20 20:53:41,456 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(952): ClusterId : a989deef-709a-42f8-a284-90450b819dbd 2023-04-20 20:53:41,462 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-20 20:53:41,471 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-20 20:53:41,471 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-20 20:53:41,475 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-20 20:53:41,476 DEBUG [RS:0;jenkins-hbase4:39223] zookeeper.ReadOnlyZKClient(139): Connect 0x7efe842c to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:41,487 DEBUG [RS:0;jenkins-hbase4:39223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70396583, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:41,488 DEBUG [RS:0;jenkins-hbase4:39223] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7406ef5c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-20 20:53:41,521 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39223 2023-04-20 20:53:41,526 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-20 20:53:41,526 INFO [RS:0;jenkins-hbase4:39223] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-20 20:53:41,527 INFO [RS:0;jenkins-hbase4:39223] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-20 20:53:41,527 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1023): About to register with Master. 2023-04-20 20:53:41,530 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,44655,1682024018782 with isa=jenkins-hbase4.apache.org/172.31.14.131:39223, startcode=1682024020002 2023-04-20 20:53:41,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-20 20:53:41,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-20 20:53:41,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-20 20:53:41,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-20 20:53:41,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-20 20:53:41,540 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,540 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-20 20:53:41,540 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,544 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682024051544 2023-04-20 20:53:41,546 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-20 20:53:41,549 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-20 20:53:41,549 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-20 20:53:41,554 DEBUG [RS:0;jenkins-hbase4:39223] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-20 20:53:41,555 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-20 20:53:41,557 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-20 20:53:41,564 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-20 20:53:41,565 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-20 20:53:41,565 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-20 20:53:41,565 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-20 20:53:41,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,569 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-20 20:53:41,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-20 20:53:41,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-20 20:53:41,575 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-20 20:53:41,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-20 20:53:41,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682024021577,5,FailOnTimeoutGroup] 2023-04-20 20:53:41,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682024021584,5,FailOnTimeoutGroup] 2023-04-20 20:53:41,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-20 20:53:41,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,598 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-20 20:53:41,600 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-20 20:53:41,601 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9 2023-04-20 20:53:41,618 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36663, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-20 20:53:41,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:41,652 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9 2023-04-20 20:53:41,652 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:34685 2023-04-20 20:53:41,652 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-20 20:53:41,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-20 20:53:41,661 DEBUG [RS:0;jenkins-hbase4:39223] zookeeper.ZKUtil(162): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:41,661 WARN [RS:0;jenkins-hbase4:39223] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-20 20:53:41,665 INFO [RS:0;jenkins-hbase4:39223] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-20 20:53:41,665 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1947): logDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/WALs/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:41,667 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39223,1682024020002] 2023-04-20 20:53:41,679 DEBUG [RS:0;jenkins-hbase4:39223] zookeeper.ZKUtil(162): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:41,692 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-20 20:53:41,704 INFO [RS:0;jenkins-hbase4:39223] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-20 20:53:41,732 INFO [RS:0;jenkins-hbase4:39223] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-20 20:53:41,739 INFO [RS:0;jenkins-hbase4:39223] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-20 20:53:41,739 INFO [RS:0;jenkins-hbase4:39223] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,740 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-20 20:53:41,751 INFO [RS:0;jenkins-hbase4:39223] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,751 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,752 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,752 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,752 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,752 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,752 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-20 20:53:41,753 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,753 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,753 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,753 DEBUG [RS:0;jenkins-hbase4:39223] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-20 20:53:41,754 INFO [RS:0;jenkins-hbase4:39223] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,754 INFO [RS:0;jenkins-hbase4:39223] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,754 INFO [RS:0;jenkins-hbase4:39223] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,775 INFO [RS:0;jenkins-hbase4:39223] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-20 20:53:41,777 INFO [RS:0;jenkins-hbase4:39223] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39223,1682024020002-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:41,798 INFO [RS:0;jenkins-hbase4:39223] regionserver.Replication(203): jenkins-hbase4.apache.org,39223,1682024020002 started 2023-04-20 20:53:41,798 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39223,1682024020002, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39223, sessionid=0x101a35999ca0001 2023-04-20 20:53:41,799 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-20 20:53:41,799 DEBUG [RS:0;jenkins-hbase4:39223] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:41,799 DEBUG [RS:0;jenkins-hbase4:39223] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39223,1682024020002' 2023-04-20 20:53:41,799 DEBUG [RS:0;jenkins-hbase4:39223] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:41,800 DEBUG [RS:0;jenkins-hbase4:39223] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:41,801 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-20 20:53:41,801 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-20 20:53:41,801 DEBUG [RS:0;jenkins-hbase4:39223] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:41,801 DEBUG [RS:0;jenkins-hbase4:39223] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39223,1682024020002' 2023-04-20 20:53:41,801 DEBUG [RS:0;jenkins-hbase4:39223] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-20 20:53:41,802 DEBUG [RS:0;jenkins-hbase4:39223] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-20 20:53:41,803 DEBUG [RS:0;jenkins-hbase4:39223] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-20 20:53:41,803 INFO [RS:0;jenkins-hbase4:39223] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-20 20:53:41,803 INFO [RS:0;jenkins-hbase4:39223] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-20 20:53:41,917 INFO [RS:0;jenkins-hbase4:39223] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39223%2C1682024020002, suffix=, logDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/WALs/jenkins-hbase4.apache.org,39223,1682024020002, archiveDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/oldWALs, maxLogs=32 2023-04-20 20:53:41,947 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42965,DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34,DISK] 2023-04-20 20:53:41,952 INFO [RS:0;jenkins-hbase4:39223] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/WALs/jenkins-hbase4.apache.org,39223,1682024020002/jenkins-hbase4.apache.org%2C39223%2C1682024020002.1682024021920 2023-04-20 20:53:41,952 DEBUG [RS:0;jenkins-hbase4:39223] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42965,DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34,DISK]] 2023-04-20 20:53:42,019 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:42,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-20 20:53:42,026 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/info 2023-04-20 20:53:42,027 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-20 20:53:42,029 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:42,029 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-20 20:53:42,032 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/rep_barrier 2023-04-20 20:53:42,033 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-20 20:53:42,034 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:42,034 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-20 20:53:42,036 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/table 2023-04-20 20:53:42,037 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-20 20:53:42,038 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:42,040 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740 2023-04-20 20:53:42,041 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740 2023-04-20 20:53:42,045 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:53:42,046 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-20 20:53:42,050 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:53:42,051 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64072772, jitterRate=-0.04524129629135132}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:53:42,051 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-20 20:53:42,051 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-20 20:53:42,051 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-20 20:53:42,051 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-20 20:53:42,051 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-20 20:53:42,051 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-20 20:53:42,052 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-20 20:53:42,053 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-20 20:53:42,057 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-20 20:53:42,057 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-20 20:53:42,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-20 20:53:42,076 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-20 20:53:42,079 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-20 20:53:42,231 DEBUG [jenkins-hbase4:44655] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-20 20:53:42,235 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39223,1682024020002, state=OPENING 2023-04-20 20:53:42,242 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-20 20:53:42,243 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:42,244 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-20 20:53:42,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:53:42,432 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:42,435 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-20 20:53:42,438 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-20 20:53:42,451 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-20 20:53:42,452 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-20 20:53:42,456 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39223%2C1682024020002.meta, suffix=.meta, logDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/WALs/jenkins-hbase4.apache.org,39223,1682024020002, archiveDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/oldWALs, maxLogs=32 2023-04-20 20:53:42,476 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42965,DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34,DISK] 2023-04-20 20:53:42,482 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/WALs/jenkins-hbase4.apache.org,39223,1682024020002/jenkins-hbase4.apache.org%2C39223%2C1682024020002.meta.1682024022458.meta 2023-04-20 20:53:42,487 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42965,DS-636cfa6c-17b3-4e9e-b53e-090cbfdcbc34,DISK]] 2023-04-20 20:53:42,488 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:53:42,491 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-20 20:53:42,512 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-20 20:53:42,518 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-20 20:53:42,525 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-20 20:53:42,525 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:42,526 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-20 20:53:42,526 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-20 20:53:42,530 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-20 20:53:42,532 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/info 2023-04-20 20:53:42,532 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/info 2023-04-20 20:53:42,532 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-20 20:53:42,533 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:42,534 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-20 20:53:42,535 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/rep_barrier 2023-04-20 20:53:42,535 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/rep_barrier 2023-04-20 20:53:42,536 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-20 20:53:42,537 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:42,537 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-20 20:53:42,538 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/table 2023-04-20 20:53:42,538 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/table 2023-04-20 20:53:42,539 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-20 20:53:42,540 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:42,542 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740 2023-04-20 20:53:42,545 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740 2023-04-20 20:53:42,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:53:42,551 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-20 20:53:42,553 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73209640, jitterRate=0.09090864658355713}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:53:42,553 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-20 20:53:42,563 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682024022424 2023-04-20 20:53:42,582 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-20 20:53:42,583 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-20 20:53:42,584 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39223,1682024020002, state=OPEN 2023-04-20 20:53:42,586 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-20 20:53:42,587 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-20 20:53:42,592 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-20 20:53:42,592 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39223,1682024020002 in 340 msec 2023-04-20 20:53:42,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-20 20:53:42,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 528 msec 2023-04-20 20:53:42,603 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1600 sec 2023-04-20 20:53:42,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682024022603, completionTime=-1 2023-04-20 20:53:42,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-20 20:53:42,604 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-20 20:53:42,676 DEBUG [hconnection-0x6bbc8bb2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:42,680 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37294, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:42,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-20 20:53:42,698 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682024082698 2023-04-20 20:53:42,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682024142699 2023-04-20 20:53:42,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 94 msec 2023-04-20 20:53:42,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44655,1682024018782-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:42,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44655,1682024018782-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:42,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44655,1682024018782-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:42,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:44655, period=300000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:42,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-20 20:53:42,734 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-20 20:53:42,744 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-20 20:53:42,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-20 20:53:42,756 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-20 20:53:42,758 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 20:53:42,761 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 20:53:42,784 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:42,786 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810 empty. 2023-04-20 20:53:42,787 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:42,787 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-20 20:53:42,812 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-20 20:53:42,815 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3d827b8c3467151cc7f52ff57fde3810, NAME => 'hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp 2023-04-20 20:53:42,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:42,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 3d827b8c3467151cc7f52ff57fde3810, disabling compactions & flushes 2023-04-20 20:53:42,833 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:53:42,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:53:42,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. after waiting 0 ms 2023-04-20 20:53:42,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:53:42,833 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:53:42,833 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 3d827b8c3467151cc7f52ff57fde3810: 2023-04-20 20:53:42,838 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 20:53:42,853 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682024022841"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024022841"}]},"ts":"1682024022841"} 2023-04-20 20:53:42,884 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 20:53:42,886 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 20:53:42,891 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024022886"}]},"ts":"1682024022886"} 2023-04-20 20:53:42,895 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-20 20:53:42,904 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3d827b8c3467151cc7f52ff57fde3810, ASSIGN}] 2023-04-20 20:53:42,907 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3d827b8c3467151cc7f52ff57fde3810, ASSIGN 2023-04-20 20:53:42,908 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=3d827b8c3467151cc7f52ff57fde3810, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=false 2023-04-20 20:53:43,059 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3d827b8c3467151cc7f52ff57fde3810, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:43,060 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682024023059"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024023059"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024023059"}]},"ts":"1682024023059"} 2023-04-20 20:53:43,064 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 3d827b8c3467151cc7f52ff57fde3810, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:53:43,225 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:53:43,225 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3d827b8c3467151cc7f52ff57fde3810, NAME => 'hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:53:43,226 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:43,227 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:43,227 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:43,227 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:43,229 INFO [StoreOpener-3d827b8c3467151cc7f52ff57fde3810-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:43,231 DEBUG [StoreOpener-3d827b8c3467151cc7f52ff57fde3810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/info 2023-04-20 20:53:43,231 DEBUG [StoreOpener-3d827b8c3467151cc7f52ff57fde3810-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/info 2023-04-20 20:53:43,232 INFO [StoreOpener-3d827b8c3467151cc7f52ff57fde3810-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d827b8c3467151cc7f52ff57fde3810 columnFamilyName info 2023-04-20 20:53:43,233 INFO [StoreOpener-3d827b8c3467151cc7f52ff57fde3810-1] regionserver.HStore(310): Store=3d827b8c3467151cc7f52ff57fde3810/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:43,235 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:43,237 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:43,242 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:53:43,246 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:53:43,247 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3d827b8c3467151cc7f52ff57fde3810; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58963362, jitterRate=-0.12137743830680847}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-20 20:53:43,247 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3d827b8c3467151cc7f52ff57fde3810: 2023-04-20 20:53:43,250 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810., pid=6, masterSystemTime=1682024023218 2023-04-20 20:53:43,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:53:43,255 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:53:43,256 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3d827b8c3467151cc7f52ff57fde3810, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:43,257 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682024023255"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024023255"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024023255"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024023255"}]},"ts":"1682024023255"} 2023-04-20 20:53:43,265 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-20 20:53:43,265 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 3d827b8c3467151cc7f52ff57fde3810, server=jenkins-hbase4.apache.org,39223,1682024020002 in 197 msec 2023-04-20 20:53:43,269 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-20 20:53:43,270 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=3d827b8c3467151cc7f52ff57fde3810, ASSIGN in 361 msec 2023-04-20 20:53:43,271 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 20:53:43,272 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024023272"}]},"ts":"1682024023272"} 2023-04-20 20:53:43,275 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-20 20:53:43,280 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 20:53:43,284 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 535 msec 2023-04-20 20:53:43,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-20 20:53:43,360 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-20 20:53:43,360 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:53:43,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-20 20:53:43,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-20 20:53:43,421 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-04-20 20:53:43,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-20 20:53:43,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-20 20:53:43,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-04-20 20:53:43,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-20 20:53:43,461 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-20 20:53:43,461 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.370sec 2023-04-20 20:53:43,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-20 20:53:43,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-20 20:53:43,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-20 20:53:43,466 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44655,1682024018782-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-20 20:53:43,467 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44655,1682024018782-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-20 20:53:43,478 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-20 20:53:43,562 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:43,567 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:43,585 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:43,598 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42990, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:43,610 INFO [Listener at localhost/45619] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:53:43,634 INFO [Listener at localhost/45619] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=357, ProcessCount=169, AvailableMemoryMB=9594 2023-04-20 20:53:43,654 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 20:53:43,657 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 20:53:43,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:53:43,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:53:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 20:53:43,676 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 20:53:43,678 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 20:53:43,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-20 20:53:43,681 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:43,683 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0 empty. 2023-04-20 20:53:43,684 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:43,685 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:53:43,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 20:53:43,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 20:53:43,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 20:53:44,101 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:53:44,104 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 061610bf32a86961a5dcdce078e5ddc0, NAME => 'TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp 2023-04-20 20:53:44,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:44,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 061610bf32a86961a5dcdce078e5ddc0, disabling compactions & flushes 2023-04-20 20:53:44,117 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. after waiting 0 ms 2023-04-20 20:53:44,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,118 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,118 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:44,123 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 20:53:44,125 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024024124"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024024124"}]},"ts":"1682024024124"} 2023-04-20 20:53:44,128 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 20:53:44,130 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 20:53:44,131 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024024130"}]},"ts":"1682024024130"} 2023-04-20 20:53:44,133 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 20:53:44,139 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=061610bf32a86961a5dcdce078e5ddc0, ASSIGN}] 2023-04-20 20:53:44,142 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=061610bf32a86961a5dcdce078e5ddc0, ASSIGN 2023-04-20 20:53:44,144 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=061610bf32a86961a5dcdce078e5ddc0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=false 2023-04-20 20:53:44,295 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=061610bf32a86961a5dcdce078e5ddc0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:44,295 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024024295"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024024295"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024024295"}]},"ts":"1682024024295"} 2023-04-20 20:53:44,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:53:44,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 20:53:44,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 061610bf32a86961a5dcdce078e5ddc0, NAME => 'TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:53:44,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:53:44,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,464 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,467 DEBUG [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A 2023-04-20 20:53:44,467 DEBUG [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A 2023-04-20 20:53:44,469 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:53:44,469 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 061610bf32a86961a5dcdce078e5ddc0 columnFamilyName A 2023-04-20 20:53:44,471 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.HStore(310): Store=061610bf32a86961a5dcdce078e5ddc0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:44,471 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,474 DEBUG [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B 2023-04-20 20:53:44,474 DEBUG [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B 2023-04-20 20:53:44,474 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:53:44,475 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 061610bf32a86961a5dcdce078e5ddc0 columnFamilyName B 2023-04-20 20:53:44,476 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.HStore(310): Store=061610bf32a86961a5dcdce078e5ddc0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:44,476 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,479 DEBUG [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C 2023-04-20 20:53:44,479 DEBUG [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C 2023-04-20 20:53:44,479 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:53:44,480 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 061610bf32a86961a5dcdce078e5ddc0 columnFamilyName C 2023-04-20 20:53:44,481 INFO [StoreOpener-061610bf32a86961a5dcdce078e5ddc0-1] regionserver.HStore(310): Store=061610bf32a86961a5dcdce078e5ddc0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:53:44,482 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:53:44,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:44,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:53:44,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 061610bf32a86961a5dcdce078e5ddc0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66850847, jitterRate=-0.0038447529077529907}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:53:44,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:44,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., pid=11, masterSystemTime=1682024024453 2023-04-20 20:53:44,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:44,504 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=061610bf32a86961a5dcdce078e5ddc0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:44,504 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024024504"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024024504"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024024504"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024024504"}]},"ts":"1682024024504"} 2023-04-20 20:53:44,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-20 20:53:44,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 in 209 msec 2023-04-20 20:53:44,516 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-20 20:53:44,517 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=061610bf32a86961a5dcdce078e5ddc0, ASSIGN in 373 msec 2023-04-20 20:53:44,518 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 20:53:44,518 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024024518"}]},"ts":"1682024024518"} 2023-04-20 20:53:44,521 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 20:53:44,524 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 20:53:44,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 852 msec 2023-04-20 20:53:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 20:53:44,804 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-20 20:53:44,809 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,817 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,820 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,829 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:44,834 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 20:53:44,837 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 20:53:44,849 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,854 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,855 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,861 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,861 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,869 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,869 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,880 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,880 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,887 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,887 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,896 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,896 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,936 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,937 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,941 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,941 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:53:44,945 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:53:44,976 DEBUG [hconnection-0x1531940c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,980 DEBUG [hconnection-0x66595227-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,980 DEBUG [hconnection-0x1e073989-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,983 DEBUG [hconnection-0x524c453b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,983 DEBUG [hconnection-0x17c8a7bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,983 DEBUG [hconnection-0x4c2fffd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,991 DEBUG [hconnection-0x26902f80-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,984 DEBUG [hconnection-0x2b3ecf8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,983 DEBUG [hconnection-0x558b6232-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:53:44,993 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:44,997 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:44,998 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43010, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:45,008 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:45,009 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:45,008 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:45,008 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:45,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:45,011 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:45,025 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:53:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:45,040 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:45,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:45,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:45,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:45,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:45,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:45,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:45,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:45,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:45,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:45,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:45,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:45,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:45,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:45,072 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:45,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:45,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:45,080 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:45,080 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:45,082 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-20 20:53:45,094 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:45,095 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:45,096 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:45,096 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:45,096 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:45,096 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:45,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:45,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:45,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024085241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024085282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024025184, max timestamp=1682024025232Num uniques 859; 2023-04-20 20:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024025184, max timestamp=1682024025232Num uniques 859; 2023-04-20 20:53:45,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-20 20:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-20 20:53:45,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024025184, max timestamp=1682024025232Num uniques 859; 2023-04-20 20:53:45,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-20 20:53:45,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024085438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024085438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024025423, max timestamp=1682024025439Num uniques 1009; 2023-04-20 20:53:45,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-20 20:53:45,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024025423, max timestamp=1682024025437Num uniques 1009; 2023-04-20 20:53:45,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-20 20:53:45,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024025423, max timestamp=1682024025439Num uniques 1009; 2023-04-20 20:53:45,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-20 20:53:45,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8090, totalHeapSize=18648, min timestamp=1682024025498, max timestamp=1682024025521Num uniques 1027; 2023-04-20 20:53:45,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47020, new segment data size=8090, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:53:45,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1682024025498, max timestamp=1682024025521Num uniques 1027; 2023-04-20 20:53:45,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=7320, suffix heap size=134776,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=160 2023-04-20 20:53:45,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024025498, max timestamp=1682024025521Num uniques 1009; 2023-04-20 20:53:45,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-20 20:53:45,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024085571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024025555, max timestamp=1682024025573Num uniques 1036; 2023-04-20 20:53:45,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-20 20:53:45,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024025555, max timestamp=1682024025573Num uniques 1036; 2023-04-20 20:53:45,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47444, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-20 20:53:45,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024025555, max timestamp=1682024025573Num uniques 1019; 2023-04-20 20:53:45,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46674, new segment data size=10806, suffix heap size=133944,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=236 2023-04-20 20:53:45,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024025602, max timestamp=1682024025625Num uniques 1095; 2023-04-20 20:53:45,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 20:53:45,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024025602, max timestamp=1682024025625Num uniques 1009; 2023-04-20 20:53:45,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46216, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-20 20:53:45,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024025602, max timestamp=1682024025627Num uniques 1095; 2023-04-20 20:53:45,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 20:53:45,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024025655, max timestamp=1682024025666Num uniques 1054; 2023-04-20 20:53:45,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=13920, suffix heap size=137584,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=304 2023-04-20 20:53:45,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024025655, max timestamp=1682024025666Num uniques 1054; 2023-04-20 20:53:45,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=13920, suffix heap size=137584,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=304 2023-04-20 20:53:45,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024025658, max timestamp=1682024025665Num uniques 1009; 2023-04-20 20:53:45,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-20 20:53:45,672 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/cfd4805581b84b02a462a41e7a79effb 2023-04-20 20:53:45,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024085721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024085724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024085725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024085733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024085733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,821 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/90cbb8a2bdb5432487b5b0cafad4c746 2023-04-20 20:53:45,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024085841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024085843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024085844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024085847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:45,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024085849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,891 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/2881c715e8d849eaa2ea029e1f702a12 2023-04-20 20:53:45,907 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/cfd4805581b84b02a462a41e7a79effb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cfd4805581b84b02a462a41e7a79effb 2023-04-20 20:53:45,919 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cfd4805581b84b02a462a41e7a79effb, entries=150, sequenceid=9, filesize=11.6 K 2023-04-20 20:53:45,929 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/90cbb8a2bdb5432487b5b0cafad4c746 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/90cbb8a2bdb5432487b5b0cafad4c746 2023-04-20 20:53:45,958 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/90cbb8a2bdb5432487b5b0cafad4c746, entries=150, sequenceid=9, filesize=11.6 K 2023-04-20 20:53:45,964 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/2881c715e8d849eaa2ea029e1f702a12 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/2881c715e8d849eaa2ea029e1f702a12 2023-04-20 20:53:45,976 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/2881c715e8d849eaa2ea029e1f702a12, entries=150, sequenceid=9, filesize=11.6 K 2023-04-20 20:53:45,981 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.42 KB/150960 for 061610bf32a86961a5dcdce078e5ddc0 in 900ms, sequenceid=9, compaction requested=false 2023-04-20 20:53:45,987 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-20 20:53:45,990 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:45,990 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:46,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:46,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c2cd38c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:46,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:46,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,009 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:46,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:46,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:46,037 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:46,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:46,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:46,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:46,037 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:46,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:46,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=154.13 KB heapSize=436.14 KB 2023-04-20 20:53:46,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:46,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:53:46,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:46,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:53:46,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:46,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:53:46,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:46,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:46,076 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,078 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:46,078 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:46,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024086068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024086077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024086082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024086084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024086104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,179 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:46,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/879fea6a840c4842b01b307b5d8c1399 2023-04-20 20:53:46,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:46,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:46,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:46,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:46,257 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:46,257 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,260 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:46,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:46,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,264 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:46,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:46,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:46,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024086210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:46,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024086210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:46,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024086211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024086210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024086263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,283 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:46,288 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:46,288 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:46,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bebb342[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:46,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:46,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:46,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:46,305 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:46,316 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:46,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:46,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:46,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:46,316 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:46,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,330 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:46,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:46,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,349 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:46,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:46,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:46,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:46,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:46,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:46,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:46,468 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:46,468 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:46,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:46,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:46,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:46,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:46,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:46,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:46,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:46,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,477 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:46,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:46,484 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:46,484 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:46,484 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:46,484 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:46,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:46,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:46,490 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,490 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70da4796[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:46,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:46,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024086483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024086484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024086487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,499 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:46,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024086487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:46,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024086488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:46,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:46,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:46,513 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:46,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:46,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:46,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:46,513 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:46,515 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,515 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,517 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,520 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:46,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,532 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:46,533 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:46,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,633 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:46,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:46,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:46,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:46,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:46,663 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:46,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:46,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:46,665 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,665 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,667 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,668 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:46,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:46,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:46,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:46,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:46,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:46,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:46,679 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:46,679 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:46,679 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:46,679 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:46,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/39fcf30ca3254658a3fbadb7a843d537 2023-04-20 20:53:46,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:46,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:46,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:46,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:46,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:46,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f42872[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:46,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:46,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:46,698 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,698 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:46,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:46,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:46,706 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:46,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:46,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:46,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:46,706 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:46,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:46,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:46,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,759 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:46,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:46,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:46,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:46,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:46,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,761 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:46,761 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:46,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:46,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:46,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.79 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/e8eb93b351774831aae63a941e646e81 2023-04-20 20:53:46,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/879fea6a840c4842b01b307b5d8c1399 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/879fea6a840c4842b01b307b5d8c1399 2023-04-20 20:53:46,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/879fea6a840c4842b01b307b5d8c1399, entries=200, sequenceid=133, filesize=14.0 K 2023-04-20 20:53:46,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/39fcf30ca3254658a3fbadb7a843d537 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/39fcf30ca3254658a3fbadb7a843d537 2023-04-20 20:53:46,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024086803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024086803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024086803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024086804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:46,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/39fcf30ca3254658a3fbadb7a843d537, entries=150, sequenceid=133, filesize=11.7 K 2023-04-20 20:53:46,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024086805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/e8eb93b351774831aae63a941e646e81 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/e8eb93b351774831aae63a941e646e81 2023-04-20 20:53:46,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/e8eb93b351774831aae63a941e646e81, entries=150, sequenceid=133, filesize=11.7 K 2023-04-20 20:53:46,835 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.13 KB/157830, heapSize ~436.09 KB/446560, currentSize=26.84 KB/27480 for 061610bf32a86961a5dcdce078e5ddc0 in 777ms, sequenceid=133, compaction requested=false 2023-04-20 20:53:46,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:46,861 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:46,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:46,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:46,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:46,880 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:46,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:46,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:46,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,890 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,891 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,891 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:46,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:46,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:46,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:46,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:46,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:46,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:46,903 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:46,903 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:46,904 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 20:53:46,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:46,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:46,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:46,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:46,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:46,907 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:46,970 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/26161f06a2fd425993093e8e231b2f25 2023-04-20 20:53:47,036 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/860498852b4a4d1daf416132c35b91ed 2023-04-20 20:53:47,090 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/43393ff1d80149e487a8e5144e3c1459 2023-04-20 20:53:47,103 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/26161f06a2fd425993093e8e231b2f25 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/26161f06a2fd425993093e8e231b2f25 2023-04-20 20:53:47,115 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/26161f06a2fd425993093e8e231b2f25, entries=100, sequenceid=140, filesize=9.4 K 2023-04-20 20:53:47,117 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/860498852b4a4d1daf416132c35b91ed as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/860498852b4a4d1daf416132c35b91ed 2023-04-20 20:53:47,133 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/860498852b4a4d1daf416132c35b91ed, entries=100, sequenceid=140, filesize=9.4 K 2023-04-20 20:53:47,135 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/43393ff1d80149e487a8e5144e3c1459 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/43393ff1d80149e487a8e5144e3c1459 2023-04-20 20:53:47,157 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/43393ff1d80149e487a8e5144e3c1459, entries=100, sequenceid=140, filesize=9.4 K 2023-04-20 20:53:47,160 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 061610bf32a86961a5dcdce078e5ddc0 in 257ms, sequenceid=140, compaction requested=true 2023-04-20 20:53:47,160 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:47,161 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:47,161 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:47,168 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725e3653[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:47,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:47,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,175 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:47,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:47,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:47,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:47,186 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:47,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:47,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:47,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:47,186 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:47,189 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,189 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:47,189 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:47,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:47,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:47,194 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:47,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:47,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:47,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:47,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:47,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:47,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:47,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:47,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:47,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:47,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:47,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:47,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:47,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:47,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:47,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:47,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:47,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:47,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:47,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:47,314 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:47,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:47,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:47,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:47,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:47,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,320 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:47,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:47,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:47,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:47,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:47,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:47,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:47,328 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:47,328 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:47,340 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:47,340 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:47,341 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 20:53:47,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:47,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:47,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:47,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:47,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:47,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:47,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:47,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:47,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024027378, max timestamp=1682024027418Num uniques 859; 2023-04-20 20:53:47,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:47,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024027378, max timestamp=1682024027418Num uniques 859; 2023-04-20 20:53:47,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:47,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024087488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024087488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024087488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024087491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024087497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024087601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024087601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024087602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024087604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024087626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024087809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024087810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024087811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024087813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024087835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:47,863 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/51203484ad2744228ce3ede64579e080 2023-04-20 20:53:47,973 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/b423963d10ca41909475d2c70d6546f4 2023-04-20 20:53:48,102 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 20:53:48,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024088115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024088117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024088115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024088117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024088149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,201 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-20 20:53:48,202 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-20 20:53:48,434 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/05d978102e134910a17dbe5a9ae492f8 2023-04-20 20:53:48,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/51203484ad2744228ce3ede64579e080 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/51203484ad2744228ce3ede64579e080 2023-04-20 20:53:48,466 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/51203484ad2744228ce3ede64579e080, entries=100, sequenceid=148, filesize=9.4 K 2023-04-20 20:53:48,469 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/b423963d10ca41909475d2c70d6546f4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b423963d10ca41909475d2c70d6546f4 2023-04-20 20:53:48,485 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b423963d10ca41909475d2c70d6546f4, entries=100, sequenceid=148, filesize=9.4 K 2023-04-20 20:53:48,495 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/05d978102e134910a17dbe5a9ae492f8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/05d978102e134910a17dbe5a9ae492f8 2023-04-20 20:53:48,511 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/05d978102e134910a17dbe5a9ae492f8, entries=100, sequenceid=148, filesize=9.4 K 2023-04-20 20:53:48,513 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=145.36 KB/148850 for 061610bf32a86961a5dcdce078e5ddc0 in 1172ms, sequenceid=148, compaction requested=true 2023-04-20 20:53:48,514 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:48,514 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:48,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:48,517 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,519 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:48,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5409faf9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:48,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:48,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,522 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:48,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:48,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:48,524 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,524 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:48,526 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:48,526 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:48,526 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:48,529 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:48,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:48,534 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:48,534 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:48,534 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:48,534 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:48,534 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:48,538 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,538 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,538 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:48,540 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:48,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:48,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:48,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:48,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:48,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:48,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:48,549 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:48,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:48,549 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:48,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:48,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=152.07 KB heapSize=443.08 KB 2023-04-20 20:53:48,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:48,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:48,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:48,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:48,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:48,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:48,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024088637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024088637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024088637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024088640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,649 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:48,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:48,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024088660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:48,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/008ae815e4c34c73a0ec9e4601de4284 2023-04-20 20:53:48,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:48,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:48,706 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:48,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:48,708 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:48,708 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:48,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,711 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:48,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:48,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:48,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:48,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:48,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:48,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:48,742 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:48,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:48,743 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:48,743 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:48,743 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:48,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:48,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,746 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024088748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024088747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76cdf119[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:48,769 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:48,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,772 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:48,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:48,775 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:48,775 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:48,775 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:48,775 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024088766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024088768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:48,779 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:48,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:48,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:48,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:48,779 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:48,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:48,782 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:48,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:48,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:48,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:48,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:48,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:48,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:48,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:48,789 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:48,789 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:48,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:48,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/b2337bc9e278403b9a9a84f9859a7ba0 2023-04-20 20:53:48,889 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:48,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:48,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:48,922 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:48,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:48,935 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:48,935 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:48,935 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,936 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-20 20:53:48,937 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-20 20:53:48,943 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:48,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:48,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:48,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:48,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:48,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:48,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:48,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:48,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:48,974 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:48,974 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:48,975 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:48,975 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:48,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:48,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024088974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:48,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024088975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:48,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f33e00a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:48,982 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:48,982 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,983 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:48,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:48,985 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:48,985 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,985 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:48,986 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:48,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024088980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:48,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:48,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024088980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:48,998 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:48,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:48,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:48,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:48,998 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:49,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,003 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,020 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:49,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:49,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,036 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:49,036 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:49,136 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:49,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:49,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:49,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:49,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:49,152 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:49,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:49,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:49,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:49,164 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:49,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:49,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,169 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:49,172 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:49,172 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:49,173 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:49,173 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:49,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:49,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:49,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:49,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:49,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:49,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:49,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cdcb511[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:49,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:49,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,193 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:49,194 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:49,196 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:49,202 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:49,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:49,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:49,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:49,202 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:49,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:49,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:49,213 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:49,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,215 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:49,215 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:49,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024089284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024089285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024089290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024089294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.56 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/490ff071b22e4832a45423146aa8cb96 2023-04-20 20:53:49,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:49,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:49,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/008ae815e4c34c73a0ec9e4601de4284 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/008ae815e4c34c73a0ec9e4601de4284 2023-04-20 20:53:49,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:49,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:49,321 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:49,321 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,323 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:49,323 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:49,324 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,324 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,325 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:49,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:49,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:49,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:49,334 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,334 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:49,335 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:49,335 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,337 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:49,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/008ae815e4c34c73a0ec9e4601de4284, entries=150, sequenceid=183, filesize=11.8 K 2023-04-20 20:53:49,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:49,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27994e41[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:49,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:49,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:49,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/b2337bc9e278403b9a9a84f9859a7ba0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b2337bc9e278403b9a9a84f9859a7ba0 2023-04-20 20:53:49,347 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:49,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:49,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:49,354 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:49,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:49,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:49,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:49,354 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:49,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b2337bc9e278403b9a9a84f9859a7ba0, entries=150, sequenceid=183, filesize=11.8 K 2023-04-20 20:53:49,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/490ff071b22e4832a45423146aa8cb96 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/490ff071b22e4832a45423146aa8cb96 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:49,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:49,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:49,373 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:49,373 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:49,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/490ff071b22e4832a45423146aa8cb96, entries=150, sequenceid=183, filesize=11.8 K 2023-04-20 20:53:49,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~443.03 KB/453664, currentSize=26.84 KB/27480 for 061610bf32a86961a5dcdce078e5ddc0 in 751ms, sequenceid=183, compaction requested=true 2023-04-20 20:53:49,385 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:49,390 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:49,390 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:49,390 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:49,390 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:53:49,391 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:53:49,406 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55192 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:53:49,408 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57582 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:53:49,409 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:53:49,409 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:53:49,409 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,409 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,409 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/90cbb8a2bdb5432487b5b0cafad4c746, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/39fcf30ca3254658a3fbadb7a843d537, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/860498852b4a4d1daf416132c35b91ed, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b423963d10ca41909475d2c70d6546f4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b2337bc9e278403b9a9a84f9859a7ba0] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=53.9 K 2023-04-20 20:53:49,410 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cfd4805581b84b02a462a41e7a79effb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/879fea6a840c4842b01b307b5d8c1399, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/26161f06a2fd425993093e8e231b2f25, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/51203484ad2744228ce3ede64579e080, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/008ae815e4c34c73a0ec9e4601de4284] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=56.2 K 2023-04-20 20:53:49,411 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting cfd4805581b84b02a462a41e7a79effb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682024025039 2023-04-20 20:53:49,411 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 90cbb8a2bdb5432487b5b0cafad4c746, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682024025039 2023-04-20 20:53:49,413 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 39fcf30ca3254658a3fbadb7a843d537, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682024025712 2023-04-20 20:53:49,413 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 879fea6a840c4842b01b307b5d8c1399, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682024025712 2023-04-20 20:53:49,414 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 860498852b4a4d1daf416132c35b91ed, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682024026076 2023-04-20 20:53:49,414 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 26161f06a2fd425993093e8e231b2f25, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682024026076 2023-04-20 20:53:49,414 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b423963d10ca41909475d2c70d6546f4, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682024027316 2023-04-20 20:53:49,415 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 51203484ad2744228ce3ede64579e080, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682024027316 2023-04-20 20:53:49,415 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b2337bc9e278403b9a9a84f9859a7ba0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682024027479 2023-04-20 20:53:49,416 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 008ae815e4c34c73a0ec9e4601de4284, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682024027479 2023-04-20 20:53:49,456 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#15 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:49,466 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#16 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:49,473 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:49,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:49,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:49,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:49,501 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:49,501 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:49,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:49,522 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,522 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,527 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:49,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:49,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:49,536 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:49,536 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:49,536 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,536 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:49,536 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 20:53:49,537 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:49,537 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:49,537 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:49,537 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:49,537 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:49,537 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:49,627 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/ed53622e1a4f4c6dacbb34a15b7b77cb 2023-04-20 20:53:49,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:49,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:49,709 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/45ff9f7dc24a4dff9a6eace3ac1cff33 2023-04-20 20:53:49,757 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/b3ec2075d0c643799d998cdce54a1d4b 2023-04-20 20:53:49,776 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/ed53622e1a4f4c6dacbb34a15b7b77cb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ed53622e1a4f4c6dacbb34a15b7b77cb 2023-04-20 20:53:49,787 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ed53622e1a4f4c6dacbb34a15b7b77cb, entries=150, sequenceid=190, filesize=11.8 K 2023-04-20 20:53:49,789 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/45ff9f7dc24a4dff9a6eace3ac1cff33 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/45ff9f7dc24a4dff9a6eace3ac1cff33 2023-04-20 20:53:49,801 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/45ff9f7dc24a4dff9a6eace3ac1cff33, entries=150, sequenceid=190, filesize=11.8 K 2023-04-20 20:53:49,803 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/b3ec2075d0c643799d998cdce54a1d4b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b3ec2075d0c643799d998cdce54a1d4b 2023-04-20 20:53:49,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024029767, max timestamp=1682024029806Num uniques 859; 2023-04-20 20:53:49,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024029767, max timestamp=1682024029807Num uniques 859; 2023-04-20 20:53:49,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:49,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:49,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024029767, max timestamp=1682024029807Num uniques 859; 2023-04-20 20:53:49,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:49,818 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b3ec2075d0c643799d998cdce54a1d4b, entries=150, sequenceid=190, filesize=11.8 K 2023-04-20 20:53:49,825 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=53.67 KB/54960 for 061610bf32a86961a5dcdce078e5ddc0 in 289ms, sequenceid=190, compaction requested=true 2023-04-20 20:53:49,825 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:49,825 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:49,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,829 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,829 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,830 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=60.38 KB heapSize=165.84 KB 2023-04-20 20:53:49,830 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:49,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:49,830 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:49,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:49,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:49,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:49,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:49,831 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:49,831 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e453692[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:49,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:49,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:49,839 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:49,848 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:49,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:49,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:49,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:49,848 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:49,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:49,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:49,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,855 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,859 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,859 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,859 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,859 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:49,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:49,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:49,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:49,860 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:49,860 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:49,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:49,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:49,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024089893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024029858, max timestamp=1682024029901Num uniques 859; 2023-04-20 20:53:49,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:53:49,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024029858, max timestamp=1682024029901Num uniques 859; 2023-04-20 20:53:49,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024029858, max timestamp=1682024029901Num uniques 859; 2023-04-20 20:53:49,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:53:49,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:53:49,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024089923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024089924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024089925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:49,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024089931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,945 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/9f8211c7e55c4d579a47715f1cd91c96 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9f8211c7e55c4d579a47715f1cd91c96 2023-04-20 20:53:49,949 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/8daf3a4bb65d42f3a0af9ebff3a461e7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8daf3a4bb65d42f3a0af9ebff3a461e7 2023-04-20 20:53:49,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:49,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:49,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:49,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:49,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:49,967 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:49,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,969 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:49,969 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:49,969 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:49,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:49,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:49,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:49,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:49,992 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,992 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:49,992 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:49,992 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:49,999 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:49,999 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:50,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e56a2e0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:50,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:50,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,006 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into 8daf3a4bb65d42f3a0af9ebff3a461e7(size=11.9 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:50,009 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into 9f8211c7e55c4d579a47715f1cd91c96(size=11.9 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:50,009 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:50,009 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=11, startTime=1682024029390; duration=0sec 2023-04-20 20:53:50,009 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:50,012 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:50,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,016 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:53:50,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:50,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:50,016 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=11, startTime=1682024029386; duration=0sec 2023-04-20 20:53:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:50,017 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:50,018 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,022 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67228 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,022 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/C is initiating minor compaction (all files) 2023-04-20 20:53:50,022 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/C in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,022 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/2881c715e8d849eaa2ea029e1f702a12, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/e8eb93b351774831aae63a941e646e81, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/43393ff1d80149e487a8e5144e3c1459, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/05d978102e134910a17dbe5a9ae492f8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/490ff071b22e4832a45423146aa8cb96, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b3ec2075d0c643799d998cdce54a1d4b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=65.7 K 2023-04-20 20:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,023 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2881c715e8d849eaa2ea029e1f702a12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682024025039 2023-04-20 20:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,024 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e8eb93b351774831aae63a941e646e81, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682024025712 2023-04-20 20:53:50,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:50,025 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 43393ff1d80149e487a8e5144e3c1459, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682024026076 2023-04-20 20:53:50,025 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:50,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:50,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:50,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:50,025 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:50,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024090005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,026 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 05d978102e134910a17dbe5a9ae492f8, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682024027316 2023-04-20 20:53:50,027 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 490ff071b22e4832a45423146aa8cb96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682024027479 2023-04-20 20:53:50,027 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b3ec2075d0c643799d998cdce54a1d4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682024028634 2023-04-20 20:53:50,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024090034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024090034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024090034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:50,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:50,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,043 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:50,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,043 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024090041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,089 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#C#compaction#21 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:50,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:50,156 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/9cf1f6d5b4aa477d977ea070d3aa97e7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9cf1f6d5b4aa477d977ea070d3aa97e7 2023-04-20 20:53:50,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:50,167 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/C of 061610bf32a86961a5dcdce078e5ddc0 into 9cf1f6d5b4aa477d977ea070d3aa97e7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:50,167 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:50,167 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/C, priority=10, startTime=1682024029390; duration=0sec 2023-04-20 20:53:50,167 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:50,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:50,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:50,172 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:50,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:50,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:50,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:50,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:50,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:50,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:50,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:50,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477bf806[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:50,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:50,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,189 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:50,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:50,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:50,200 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:50,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:50,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:50,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:50,201 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:50,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,203 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:50,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:50,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,210 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:50,210 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:50,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,211 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,211 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:50,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024090229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024090241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024090241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024090241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024090253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/bf5840d21d68484f8733e839ed61ae44 2023-04-20 20:53:50,310 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:50,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:50,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:50,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:50,344 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:50,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:50,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:50,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,348 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:50,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:50,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:50,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,368 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:50,369 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:50,369 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:50,369 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,369 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:50,369 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:50,370 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a7d4dee[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:50,390 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:50,390 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/69a52f6ff4714ddda20fbfc2b16f8ba3 2023-04-20 20:53:50,392 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:50,394 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,394 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:50,395 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:50,395 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,395 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:50,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:50,415 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:50,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:50,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:50,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:50,415 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:50,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,421 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,423 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,427 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,427 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,427 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:50,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:50,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:50,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,428 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:50,428 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:50,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,528 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:50,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:50,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:50,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:50,534 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:50,534 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:50,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:50,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024090532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:50,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:50,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:50,541 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:50,541 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:50,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:50,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:50,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:50,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024090545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:50,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024090545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d1b1f95[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:50,549 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:50,549 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024090546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,555 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:50,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:50,557 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024090558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:50,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:50,562 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:50,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:50,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:50,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:50,562 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:50,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,564 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:50,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:50,570 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:50,570 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:50,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:50,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,670 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:50,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:50,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:50,679 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:50,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:50,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:50,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,684 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:50,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:50,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:50,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:50,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:50,692 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,694 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,694 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:50,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1671eb74[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:50,694 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:50,694 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,696 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:50,696 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:50,696 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:50,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:50,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:50,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:50,702 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:50,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:50,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:50,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:50,702 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:50,705 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,705 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,705 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:50,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:50,732 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:50,732 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:50,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,832 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:50,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:50,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:50,837 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:50,837 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:50,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:50,839 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,839 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,840 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:50,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:50,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:50,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:50,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:50,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:50,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/23a107b66d8f4a0b871c61c1af9ede42 2023-04-20 20:53:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,846 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed28a74[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:50,849 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:50,849 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:50,852 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:50,854 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/bf5840d21d68484f8733e839ed61ae44 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/bf5840d21d68484f8733e839ed61ae44 2023-04-20 20:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:50,860 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:50,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:50,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:50,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:50,860 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:50,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,866 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:50,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:50,869 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:50,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:50,869 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:50,871 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:50,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:50,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:50,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/bf5840d21d68484f8733e839ed61ae44, entries=250, sequenceid=216, filesize=16.4 K 2023-04-20 20:53:50,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/69a52f6ff4714ddda20fbfc2b16f8ba3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69a52f6ff4714ddda20fbfc2b16f8ba3 2023-04-20 20:53:50,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69a52f6ff4714ddda20fbfc2b16f8ba3, entries=150, sequenceid=216, filesize=11.8 K 2023-04-20 20:53:50,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/23a107b66d8f4a0b871c61c1af9ede42 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/23a107b66d8f4a0b871c61c1af9ede42 2023-04-20 20:53:50,896 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/23a107b66d8f4a0b871c61c1af9ede42, entries=150, sequenceid=216, filesize=11.8 K 2023-04-20 20:53:50,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=127.47 KB/130530 for 061610bf32a86961a5dcdce078e5ddc0 in 1068ms, sequenceid=216, compaction requested=true 2023-04-20 20:53:50,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:50,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:50,898 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:53:50,899 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:53:50,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:50,899 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:50,900 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41060 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:53:50,900 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:53:50,901 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:53:50,901 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:53:50,901 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,901 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,901 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9f8211c7e55c4d579a47715f1cd91c96, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ed53622e1a4f4c6dacbb34a15b7b77cb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/bf5840d21d68484f8733e839ed61ae44] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=40.1 K 2023-04-20 20:53:50,901 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8daf3a4bb65d42f3a0af9ebff3a461e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/45ff9f7dc24a4dff9a6eace3ac1cff33, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69a52f6ff4714ddda20fbfc2b16f8ba3] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=35.4 K 2023-04-20 20:53:50,902 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9f8211c7e55c4d579a47715f1cd91c96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682024027479 2023-04-20 20:53:50,902 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8daf3a4bb65d42f3a0af9ebff3a461e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682024027479 2023-04-20 20:53:50,905 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ed53622e1a4f4c6dacbb34a15b7b77cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682024028634 2023-04-20 20:53:50,905 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 45ff9f7dc24a4dff9a6eace3ac1cff33, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682024028634 2023-04-20 20:53:50,905 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting bf5840d21d68484f8733e839ed61ae44, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682024029767 2023-04-20 20:53:50,906 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 69a52f6ff4714ddda20fbfc2b16f8ba3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682024029813 2023-04-20 20:53:50,928 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#24 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:50,931 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#25 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:50,971 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:50,973 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/85d14189d1fc426186e66b7f40436881 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/85d14189d1fc426186e66b7f40436881 2023-04-20 20:53:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:50,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:50,983 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into 85d14189d1fc426186e66b7f40436881(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:50,983 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:50,983 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=13, startTime=1682024030898; duration=0sec 2023-04-20 20:53:50,983 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:50,984 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-20 20:53:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:50,985 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:50,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,986 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:50,986 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:50,986 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,987 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,987 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 20:53:50,987 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 20:53:50,987 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. because compaction request was cancelled 2023-04-20 20:53:50,989 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,989 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:50,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,990 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:50,990 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:50,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:50,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:50,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:50,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:50,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:50,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:50,997 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-04-20 20:53:50,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:50,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:50,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:50,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:50,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:50,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:51,022 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/a688a4a7dfcd4538a22576f071cfa27c 2023-04-20 20:53:51,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:51,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:51,049 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/27d37e1c38b34f859a8f5ab54a6cf7bb 2023-04-20 20:53:51,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024091065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024091066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024091066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024091070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024091071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024091177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024091181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024091182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024091183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,374 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/cf8034b45e434df7b7f008f296e83e3c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cf8034b45e434df7b7f008f296e83e3c 2023-04-20 20:53:51,384 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into cf8034b45e434df7b7f008f296e83e3c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:51,384 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:51,384 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=13, startTime=1682024030898; duration=0sec 2023-04-20 20:53:51,384 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:51,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024091385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024091389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024091389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024091390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,485 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/8ded717aa5a14384a8b0a11d6440a652 2023-04-20 20:53:51,496 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/a688a4a7dfcd4538a22576f071cfa27c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a688a4a7dfcd4538a22576f071cfa27c 2023-04-20 20:53:51,506 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a688a4a7dfcd4538a22576f071cfa27c, entries=150, sequenceid=253, filesize=11.8 K 2023-04-20 20:53:51,509 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/27d37e1c38b34f859a8f5ab54a6cf7bb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/27d37e1c38b34f859a8f5ab54a6cf7bb 2023-04-20 20:53:51,521 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/27d37e1c38b34f859a8f5ab54a6cf7bb, entries=150, sequenceid=253, filesize=11.8 K 2023-04-20 20:53:51,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/8ded717aa5a14384a8b0a11d6440a652 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/8ded717aa5a14384a8b0a11d6440a652 2023-04-20 20:53:51,536 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/8ded717aa5a14384a8b0a11d6440a652, entries=150, sequenceid=253, filesize=11.8 K 2023-04-20 20:53:51,539 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for 061610bf32a86961a5dcdce078e5ddc0 in 542ms, sequenceid=253, compaction requested=true 2023-04-20 20:53:51,539 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:51,539 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:51,542 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,542 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:51,542 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,542 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e00b47d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:51,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:51,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:51,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:51,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:51,554 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:51,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:51,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:51,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:51,554 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:51,560 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:51,560 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,560 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:51,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:51,563 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:51,567 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:51,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:51,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:51,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:51,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:51,570 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:51,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:51,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:51,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:51,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:51,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:51,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:51,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:51,671 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:51,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:51,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:51,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:51,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:51,679 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:51,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:51,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:51,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:51,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:51,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:51,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:51,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:51,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:51,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:51,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:51,686 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:51,686 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:51,687 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:53:51,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:51,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:51,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:51,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:51,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:51,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:51,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:51,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:51,714 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/da8130d7df964651bb834856ca8d3eb7 2023-04-20 20:53:51,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024091717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024091719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024091721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024091721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024091823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024091823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024091825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:51,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:51,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024091825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024092028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024092028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024092028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024092029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024092082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,133 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/31d158c884f64242810c190c4c59e460 2023-04-20 20:53:52,186 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/052fe2f9a0b94d089136c6740f687fe0 2023-04-20 20:53:52,197 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/da8130d7df964651bb834856ca8d3eb7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/da8130d7df964651bb834856ca8d3eb7 2023-04-20 20:53:52,207 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/da8130d7df964651bb834856ca8d3eb7, entries=150, sequenceid=266, filesize=11.9 K 2023-04-20 20:53:52,211 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/31d158c884f64242810c190c4c59e460 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/31d158c884f64242810c190c4c59e460 2023-04-20 20:53:52,221 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/31d158c884f64242810c190c4c59e460, entries=150, sequenceid=266, filesize=11.9 K 2023-04-20 20:53:52,228 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/052fe2f9a0b94d089136c6740f687fe0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/052fe2f9a0b94d089136c6740f687fe0 2023-04-20 20:53:52,246 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/052fe2f9a0b94d089136c6740f687fe0, entries=150, sequenceid=266, filesize=11.9 K 2023-04-20 20:53:52,249 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 061610bf32a86961a5dcdce078e5ddc0 in 562ms, sequenceid=266, compaction requested=true 2023-04-20 20:53:52,249 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:52,249 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,249 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:52,254 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b82ed9f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:52,257 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:52,257 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,259 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:52,260 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,260 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:52,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:52,261 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:52,266 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:52,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:52,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:52,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:52,266 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:52,268 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,268 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,268 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:52,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:52,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,274 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:52,275 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:52,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024031702, max timestamp=1682024032357Num uniques 859; 2023-04-20 20:53:52,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:52,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:52,361 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=96.16 KB heapSize=282.03 KB 2023-04-20 20:53:52,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024031702, max timestamp=1682024032359Num uniques 859; 2023-04-20 20:53:52,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:52,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024031702, max timestamp=1682024032359Num uniques 859; 2023-04-20 20:53:52,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:52,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:52,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:52,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:52,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:52,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:52,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:52,376 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:52,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:52,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:52,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:52,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:52,392 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:52,392 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:52,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:52,397 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,397 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,400 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,400 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,400 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,400 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:52,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:52,415 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:52,415 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:52,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024032381, max timestamp=1682024032414Num uniques 859; 2023-04-20 20:53:52,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:53:52,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682024032381, max timestamp=1682024032414Num uniques 859; 2023-04-20 20:53:52,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:53:52,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682024032381, max timestamp=1682024032414Num uniques 859; 2023-04-20 20:53:52,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:53:52,422 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:52,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:52,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,431 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b189d0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:52,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:52,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:52,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:52,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/34cae5e9034045479e43ea6bcaceceac 2023-04-20 20:53:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:52,451 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:52,451 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:52,451 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:52,451 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:52,451 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:52,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,458 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,461 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,462 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:52,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:52,463 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:52,463 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:52,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024092456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024092457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024092463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024092465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,563 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:52,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:52,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024092566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:52,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024092570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:52,571 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:52,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:52,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:52,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024092571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,572 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,572 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,575 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,576 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,576 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:52,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:52,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:52,580 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:52,580 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:52,580 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,580 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:52,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024092577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,582 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:52,582 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:52,584 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:52,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282a24d8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:52,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:52,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,589 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:52,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:52,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:52,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:52,592 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:52,596 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:52,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:52,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:52,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:52,596 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:52,598 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,598 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,598 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,601 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:52,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:52,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,612 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,612 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:52,612 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,712 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:52,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:52,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:52,716 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:52,716 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,716 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:52,716 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:52,717 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,717 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,718 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:52,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:52,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:52,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:52,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:52,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:52,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:52,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63c2defc[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:52,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:52,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,726 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:52,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:52,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:52,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:52,728 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:52,733 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:52,733 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:52,733 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:52,733 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:52,733 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:52,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:52,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:52,752 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,752 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:52,752 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:52,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024092772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024092774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024092774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:52,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024092783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,852 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:52,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:52,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:52,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:52,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:52,859 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:52,859 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,859 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:52,859 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:52,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:52,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:52,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:52,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:52,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:52,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:52,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:52,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:52,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:52,865 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:52,865 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:52,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,866 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:52,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f10290e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:52,868 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:52,868 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:52,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:52,870 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,870 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:52,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,871 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,871 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,871 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:52,876 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:52,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:52,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:52,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:52,876 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:52,879 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,879 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,879 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:52,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:52,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/d691c9edb86349189fa577999b9443ec 2023-04-20 20:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,885 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,891 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:52,891 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:52,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:52,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:52,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:52,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:52,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:52,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:52,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:52,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:52,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:52,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:52,901 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:52,901 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:52,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/824712ac404b41b7a56ffecbcaca12cd 2023-04-20 20:53:52,945 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/34cae5e9034045479e43ea6bcaceceac as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/34cae5e9034045479e43ea6bcaceceac 2023-04-20 20:53:52,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/34cae5e9034045479e43ea6bcaceceac, entries=300, sequenceid=289, filesize=19.0 K 2023-04-20 20:53:52,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/d691c9edb86349189fa577999b9443ec as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d691c9edb86349189fa577999b9443ec 2023-04-20 20:53:52,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d691c9edb86349189fa577999b9443ec, entries=150, sequenceid=289, filesize=11.9 K 2023-04-20 20:53:52,986 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/824712ac404b41b7a56ffecbcaca12cd as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/824712ac404b41b7a56ffecbcaca12cd 2023-04-20 20:53:52,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/824712ac404b41b7a56ffecbcaca12cd, entries=150, sequenceid=289, filesize=11.9 K 2023-04-20 20:53:53,000 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=147.60 KB/151140 for 061610bf32a86961a5dcdce078e5ddc0 in 639ms, sequenceid=289, compaction requested=true 2023-04-20 20:53:53,001 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,001 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:53,001 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:53,001 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:53:53,001 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:53:53,001 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:53,001 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:53,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:53,004 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48718 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:53:53,004 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:53:53,004 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,004 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/85d14189d1fc426186e66b7f40436881, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/27d37e1c38b34f859a8f5ab54a6cf7bb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/31d158c884f64242810c190c4c59e460, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d691c9edb86349189fa577999b9443ec] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=47.6 K 2023-04-20 20:53:53,004 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:53:53,004 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:53:53,005 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:53,006 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cf8034b45e434df7b7f008f296e83e3c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a688a4a7dfcd4538a22576f071cfa27c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/da8130d7df964651bb834856ca8d3eb7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/34cae5e9034045479e43ea6bcaceceac] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=54.7 K 2023-04-20 20:53:53,007 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 85d14189d1fc426186e66b7f40436881, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682024029813 2023-04-20 20:53:53,008 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting cf8034b45e434df7b7f008f296e83e3c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682024029813 2023-04-20 20:53:53,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:53,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:53,009 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:53,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:53,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:53,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:53,009 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a688a4a7dfcd4538a22576f071cfa27c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682024029921 2023-04-20 20:53:53,009 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 27d37e1c38b34f859a8f5ab54a6cf7bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682024029921 2023-04-20 20:53:53,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,010 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 31d158c884f64242810c190c4c59e460, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682024031058 2023-04-20 20:53:53,010 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting da8130d7df964651bb834856ca8d3eb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682024031058 2023-04-20 20:53:53,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,011 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d691c9edb86349189fa577999b9443ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682024032357 2023-04-20 20:53:53,011 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 34cae5e9034045479e43ea6bcaceceac, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682024031702 2023-04-20 20:53:53,012 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:53,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:53,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:53,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:53,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:53,021 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,021 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:53,021 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-20 20:53:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:53,042 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:53,050 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:53,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:53,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:53,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024093092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024093095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024093094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43092 deadline: 1682024093096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,102 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/28cd84d83b874aafb3f1d85e260874ce as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/28cd84d83b874aafb3f1d85e260874ce 2023-04-20 20:53:53,113 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/84fd3287138541049b217293d4ceed3f 2023-04-20 20:53:53,118 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into 28cd84d83b874aafb3f1d85e260874ce(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:53,119 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,119 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=12, startTime=1682024033001; duration=0sec 2023-04-20 20:53:53,119 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:53,120 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:53:53,124 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60686 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:53:53,124 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/C is initiating minor compaction (all files) 2023-04-20 20:53:53,124 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/C in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,124 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9cf1f6d5b4aa477d977ea070d3aa97e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/23a107b66d8f4a0b871c61c1af9ede42, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/8ded717aa5a14384a8b0a11d6440a652, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/052fe2f9a0b94d089136c6740f687fe0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/824712ac404b41b7a56ffecbcaca12cd] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=59.3 K 2023-04-20 20:53:53,125 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9cf1f6d5b4aa477d977ea070d3aa97e7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682024028634 2023-04-20 20:53:53,125 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 23a107b66d8f4a0b871c61c1af9ede42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682024029813 2023-04-20 20:53:53,126 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8ded717aa5a14384a8b0a11d6440a652, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1682024029921 2023-04-20 20:53:53,127 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 052fe2f9a0b94d089136c6740f687fe0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682024031058 2023-04-20 20:53:53,128 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 824712ac404b41b7a56ffecbcaca12cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682024032357 2023-04-20 20:53:53,170 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#C#compaction#39 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:53,186 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/69f4740c9cd94e2bab42f34474aff4ff as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/69f4740c9cd94e2bab42f34474aff4ff 2023-04-20 20:53:53,193 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/C of 061610bf32a86961a5dcdce078e5ddc0 into 69f4740c9cd94e2bab42f34474aff4ff(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:53,193 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,194 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/C, priority=11, startTime=1682024033001; duration=0sec 2023-04-20 20:53:53,194 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:53,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024093200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024093201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024093203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43044 deadline: 1682024093404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43078 deadline: 1682024093406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024093410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,483 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/9dc342cda4e04d748b6dc50656b2548e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9dc342cda4e04d748b6dc50656b2548e 2023-04-20 20:53:53,492 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into 9dc342cda4e04d748b6dc50656b2548e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:53,492 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,492 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=12, startTime=1682024033001; duration=0sec 2023-04-20 20:53:53,493 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:53,548 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/cc3b1c84540a4c8fa8d230f855a15e4b 2023-04-20 20:53:53,579 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/06a4527abf8940a9a938c2a6a5d1cdac 2023-04-20 20:53:53,589 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/84fd3287138541049b217293d4ceed3f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/84fd3287138541049b217293d4ceed3f 2023-04-20 20:53:53,597 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/84fd3287138541049b217293d4ceed3f, entries=150, sequenceid=325, filesize=11.9 K 2023-04-20 20:53:53,598 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/cc3b1c84540a4c8fa8d230f855a15e4b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/cc3b1c84540a4c8fa8d230f855a15e4b 2023-04-20 20:53:53,604 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/cc3b1c84540a4c8fa8d230f855a15e4b, entries=150, sequenceid=325, filesize=11.9 K 2023-04-20 20:53:53,605 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/06a4527abf8940a9a938c2a6a5d1cdac as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/06a4527abf8940a9a938c2a6a5d1cdac 2023-04-20 20:53:53,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024093605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,615 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/06a4527abf8940a9a938c2a6a5d1cdac, entries=150, sequenceid=325, filesize=11.9 K 2023-04-20 20:53:53,617 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 061610bf32a86961a5dcdce078e5ddc0 in 596ms, sequenceid=325, compaction requested=false 2023-04-20 20:53:53,617 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,617 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:53,619 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,619 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6151d435[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:53,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:53,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:53,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:53,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:53,631 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:53,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:53,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:53,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:53,631 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:53,633 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,633 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,633 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:53,634 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,648 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,648 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:53,648 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:53,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:53,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:53,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:53,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,649 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:53,649 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:53,649 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:53,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:53,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:53,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:53,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:53:53,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:53,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:53,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:53,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:53,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:53,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:53,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/6ff7b6cfd354487989dbaec517794b1d 2023-04-20 20:53:53,749 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:53,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:53,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024033740, max timestamp=1682024033758Num uniques 859; 2023-04-20 20:53:53,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024033740, max timestamp=1682024033758Num uniques 859; 2023-04-20 20:53:53,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024033740, max timestamp=1682024033758Num uniques 859; 2023-04-20 20:53:53,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:53,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:53,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:53:53,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:53,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:53,763 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:53,763 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:53,763 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:53,763 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:53,764 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/1453848fa21e49a892bbbc2f6da7ef1a 2023-04-20 20:53:53,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:53,767 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:53,767 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,767 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:53,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:53,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:53,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:53,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:53,770 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,770 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:53,770 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:53,770 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,772 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,774 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,774 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:53,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18242023[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:53,776 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:53,776 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,778 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:53,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:53,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,780 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:53,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:53,786 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:53,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:53,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:53,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:53,786 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:53,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/993aa8898eb648bab6182da846059bca 2023-04-20 20:53:53,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:53,791 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,793 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/6ff7b6cfd354487989dbaec517794b1d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6ff7b6cfd354487989dbaec517794b1d 2023-04-20 20:53:53,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:53,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:53,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:53,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:53,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:53,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,798 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:53,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:53,798 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,798 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,798 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:53,798 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:53,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:53,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:53,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024093799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024033787, max timestamp=1682024033799Num uniques 1018; 2023-04-20 20:53:53,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024033787, max timestamp=1682024033799Num uniques 1018; 2023-04-20 20:53:53,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 20:53:53,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 20:53:53,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024033787, max timestamp=1682024033799Num uniques 1018; 2023-04-20 20:53:53,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 20:53:53,804 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6ff7b6cfd354487989dbaec517794b1d, entries=200, sequenceid=338, filesize=14.3 K 2023-04-20 20:53:53,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/1453848fa21e49a892bbbc2f6da7ef1a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/1453848fa21e49a892bbbc2f6da7ef1a 2023-04-20 20:53:53,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/1453848fa21e49a892bbbc2f6da7ef1a, entries=150, sequenceid=338, filesize=11.9 K 2023-04-20 20:53:53,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/993aa8898eb648bab6182da846059bca as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/993aa8898eb648bab6182da846059bca 2023-04-20 20:53:53,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/993aa8898eb648bab6182da846059bca, entries=150, sequenceid=338, filesize=11.9 K 2023-04-20 20:53:53,827 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=60.38 KB/61830 for 061610bf32a86961a5dcdce078e5ddc0 in 117ms, sequenceid=338, compaction requested=true 2023-04-20 20:53:53,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,827 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:53,827 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:53:53,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:53,828 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:53:53,836 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39408 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:53:53,836 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:53:53,836 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,837 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9dc342cda4e04d748b6dc50656b2548e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/84fd3287138541049b217293d4ceed3f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6ff7b6cfd354487989dbaec517794b1d] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=38.5 K 2023-04-20 20:53:53,837 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:53:53,837 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:53:53,837 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,837 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/28cd84d83b874aafb3f1d85e260874ce, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/cc3b1c84540a4c8fa8d230f855a15e4b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/1453848fa21e49a892bbbc2f6da7ef1a] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=36.1 K 2023-04-20 20:53:53,827 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:53,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=80.51 KB heapSize=224.77 KB 2023-04-20 20:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:53,848 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:53:53,848 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9dc342cda4e04d748b6dc50656b2548e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682024032357 2023-04-20 20:53:53,848 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 28cd84d83b874aafb3f1d85e260874ce, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682024032357 2023-04-20 20:53:53,854 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 84fd3287138541049b217293d4ceed3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1682024032452 2023-04-20 20:53:53,854 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting cc3b1c84540a4c8fa8d230f855a15e4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1682024032452 2023-04-20 20:53:53,854 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6ff7b6cfd354487989dbaec517794b1d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682024033090 2023-04-20 20:53:53,855 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1453848fa21e49a892bbbc2f6da7ef1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682024033090 2023-04-20 20:53:53,897 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:53,898 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:53,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:53,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/754664cb4ce644a08e08ff28ad96fcd9 2023-04-20 20:53:53,901 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:53,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:53,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024093910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024093911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:53,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024093912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:53,916 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:53,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:53,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:53,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:53,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:53,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:53,921 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:53,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:53,928 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:53,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:53,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:53,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:53,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:53,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6438e5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:53,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:53,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,942 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:53,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:53,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:53,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:53,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:53,952 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:53,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:53,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:53,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:53,952 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:53,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,958 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/b26c34664c1049b196be5403d6560d97 2023-04-20 20:53:53,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:53,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:53,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:53,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:53,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:53,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:53,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:53,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:53,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:53,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:53,971 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:53,971 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:53,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:53,972 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/57f784f84a6e4b728d4a47158025390a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/57f784f84a6e4b728d4a47158025390a 2023-04-20 20:53:53,979 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/2201b1a8f5d24251b1ac1c822f50496d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2201b1a8f5d24251b1ac1c822f50496d 2023-04-20 20:53:53,987 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into 57f784f84a6e4b728d4a47158025390a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:53,987 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,987 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=13, startTime=1682024033827; duration=0sec 2023-04-20 20:53:53,987 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:53,987 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:53:53,989 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:53:53,989 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/C is initiating minor compaction (all files) 2023-04-20 20:53:53,990 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/C in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:53,990 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/69f4740c9cd94e2bab42f34474aff4ff, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/06a4527abf8940a9a938c2a6a5d1cdac, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/993aa8898eb648bab6182da846059bca] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=36.1 K 2023-04-20 20:53:53,990 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into 2201b1a8f5d24251b1ac1c822f50496d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:53,990 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:53,990 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=13, startTime=1682024033827; duration=0sec 2023-04-20 20:53:53,990 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:53,990 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 69f4740c9cd94e2bab42f34474aff4ff, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682024032357 2023-04-20 20:53:53,993 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 06a4527abf8940a9a938c2a6a5d1cdac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1682024032452 2023-04-20 20:53:53,994 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 993aa8898eb648bab6182da846059bca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682024033090 2023-04-20 20:53:54,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/893353891b104f0e9873478e06297be8 2023-04-20 20:53:54,005 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#C#compaction#49 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:54,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/754664cb4ce644a08e08ff28ad96fcd9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/754664cb4ce644a08e08ff28ad96fcd9 2023-04-20 20:53:54,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/754664cb4ce644a08e08ff28ad96fcd9, entries=150, sequenceid=384, filesize=11.9 K 2023-04-20 20:53:54,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024094017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/b26c34664c1049b196be5403d6560d97 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b26c34664c1049b196be5403d6560d97 2023-04-20 20:53:54,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024094017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024094018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b26c34664c1049b196be5403d6560d97, entries=150, sequenceid=384, filesize=11.9 K 2023-04-20 20:53:54,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/893353891b104f0e9873478e06297be8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/893353891b104f0e9873478e06297be8 2023-04-20 20:53:54,044 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/f8a2f94e0e4e49b4b0a5050c502338f9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/f8a2f94e0e4e49b4b0a5050c502338f9 2023-04-20 20:53:54,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/893353891b104f0e9873478e06297be8, entries=150, sequenceid=384, filesize=11.9 K 2023-04-20 20:53:54,049 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~224.72 KB/230112, currentSize=100.63 KB/103050 for 061610bf32a86961a5dcdce078e5ddc0 in 209ms, sequenceid=384, compaction requested=false 2023-04-20 20:53:54,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:54,055 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/C of 061610bf32a86961a5dcdce078e5ddc0 into f8a2f94e0e4e49b4b0a5050c502338f9(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:54,056 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:54,056 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/C, priority=13, startTime=1682024033840; duration=0sec 2023-04-20 20:53:54,056 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:54,071 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:54,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:54,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:54,080 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:54,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:54,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:54,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:54,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:54,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:54,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:54,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:54,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:54,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:54,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:54,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:54,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:54,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:54,088 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:54,088 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:54,088 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 20:53:54,088 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:54,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:54,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:54,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:54,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:54,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:54,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:54,097 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:54,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024094148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024094221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024094221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024094234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024094257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024094460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,502 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/0a8584d1d6e547d0876608ed6db253d5 2023-04-20 20:53:54,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024094524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024094525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024094538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024094615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024094764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:54,920 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/f2414b1f85044dc185c9e2c5d7fd40e7 2023-04-20 20:53:55,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024095031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024095032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024095040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:55,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024095276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,344 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/62c49e92df684b0ebeba9a0b3f02c8ea 2023-04-20 20:53:55,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/0a8584d1d6e547d0876608ed6db253d5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0a8584d1d6e547d0876608ed6db253d5 2023-04-20 20:53:55,357 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0a8584d1d6e547d0876608ed6db253d5, entries=150, sequenceid=405, filesize=11.9 K 2023-04-20 20:53:55,359 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/f2414b1f85044dc185c9e2c5d7fd40e7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f2414b1f85044dc185c9e2c5d7fd40e7 2023-04-20 20:53:55,380 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f2414b1f85044dc185c9e2c5d7fd40e7, entries=150, sequenceid=405, filesize=11.9 K 2023-04-20 20:53:55,381 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/62c49e92df684b0ebeba9a0b3f02c8ea as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/62c49e92df684b0ebeba9a0b3f02c8ea 2023-04-20 20:53:55,390 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/62c49e92df684b0ebeba9a0b3f02c8ea, entries=150, sequenceid=405, filesize=11.9 K 2023-04-20 20:53:55,391 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 061610bf32a86961a5dcdce078e5ddc0 in 1303ms, sequenceid=405, compaction requested=true 2023-04-20 20:53:55,391 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:55,391 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:55,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e2241e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:55,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:55,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,510 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:55,511 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,511 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:55,610 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:55,618 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:55,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:55,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:55,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:55,619 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:55,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:55,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:55,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:55,692 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,764 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,764 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:55,765 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:55,765 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:55,765 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:55,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:55,765 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:55,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:55,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:55,765 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,765 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:55,765 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,765 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:55,765 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:55,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:55,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:55,866 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:55,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:55,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:55,879 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:55,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:55,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:55,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:55,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:55,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:55,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:55,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:55,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:55,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:55,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:55,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:55,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:55,912 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:55,912 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:55,912 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 20:53:55,912 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:55,913 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:55,913 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:55,913 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:55,913 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:55,913 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:55,960 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/41d0c8b4589e4000bd41a78cc8ad3d44 2023-04-20 20:53:56,022 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/d96d9939504c4e4197a427dfb5d8a316 2023-04-20 20:53:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:56,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:56,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024096069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,074 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/dd7918b719504002806890c227ba680d 2023-04-20 20:53:56,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024096071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024096074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/41d0c8b4589e4000bd41a78cc8ad3d44 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/41d0c8b4589e4000bd41a78cc8ad3d44 2023-04-20 20:53:56,092 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/41d0c8b4589e4000bd41a78cc8ad3d44, entries=150, sequenceid=419, filesize=11.9 K 2023-04-20 20:53:56,093 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/d96d9939504c4e4197a427dfb5d8a316 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d96d9939504c4e4197a427dfb5d8a316 2023-04-20 20:53:56,100 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d96d9939504c4e4197a427dfb5d8a316, entries=150, sequenceid=419, filesize=11.9 K 2023-04-20 20:53:56,101 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/dd7918b719504002806890c227ba680d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/dd7918b719504002806890c227ba680d 2023-04-20 20:53:56,109 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/dd7918b719504002806890c227ba680d, entries=150, sequenceid=419, filesize=11.9 K 2023-04-20 20:53:56,111 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 061610bf32a86961a5dcdce078e5ddc0 in 199ms, sequenceid=419, compaction requested=true 2023-04-20 20:53:56,111 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:56,111 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:56,117 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:56,117 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,117 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:56,117 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:56,180 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 20:53:56,180 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:56,180 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:56,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:56,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:56,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:56,251 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c7dae2b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:56,257 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:56,257 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024096247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024096247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024096248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/169db961cd9d4c0ebf88bfdfef2812bd 2023-04-20 20:53:56,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/154754891409478e9c933a352f958ca4 2023-04-20 20:53:56,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024096282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,318 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,318 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,318 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:56,318 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:56,318 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:56,318 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:56,320 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/1eefe1566f914e2aadbdae23166cb091 2023-04-20 20:53:56,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/169db961cd9d4c0ebf88bfdfef2812bd as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/169db961cd9d4c0ebf88bfdfef2812bd 2023-04-20 20:53:56,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/169db961cd9d4c0ebf88bfdfef2812bd, entries=150, sequenceid=438, filesize=11.9 K 2023-04-20 20:53:56,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/154754891409478e9c933a352f958ca4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/154754891409478e9c933a352f958ca4 2023-04-20 20:53:56,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/154754891409478e9c933a352f958ca4, entries=150, sequenceid=438, filesize=11.9 K 2023-04-20 20:53:56,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/1eefe1566f914e2aadbdae23166cb091 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/1eefe1566f914e2aadbdae23166cb091 2023-04-20 20:53:56,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/1eefe1566f914e2aadbdae23166cb091, entries=150, sequenceid=438, filesize=11.9 K 2023-04-20 20:53:56,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 061610bf32a86961a5dcdce078e5ddc0 in 167ms, sequenceid=438, compaction requested=true 2023-04-20 20:53:56,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:56,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:56,348 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:53:56,348 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:53:56,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:56,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:56,351 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:53:56,351 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:53:56,351 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:53:56,351 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:53:56,351 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:56,351 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:56,351 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/57f784f84a6e4b728d4a47158025390a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b26c34664c1049b196be5403d6560d97, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f2414b1f85044dc185c9e2c5d7fd40e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d96d9939504c4e4197a427dfb5d8a316, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/154754891409478e9c933a352f958ca4] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=60.0 K 2023-04-20 20:53:56,351 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2201b1a8f5d24251b1ac1c822f50496d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/754664cb4ce644a08e08ff28ad96fcd9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0a8584d1d6e547d0876608ed6db253d5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/41d0c8b4589e4000bd41a78cc8ad3d44, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/169db961cd9d4c0ebf88bfdfef2812bd] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=60.0 K 2023-04-20 20:53:56,352 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 57f784f84a6e4b728d4a47158025390a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682024033090 2023-04-20 20:53:56,353 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2201b1a8f5d24251b1ac1c822f50496d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682024033090 2023-04-20 20:53:56,354 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b26c34664c1049b196be5403d6560d97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682024033812 2023-04-20 20:53:56,355 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 754664cb4ce644a08e08ff28ad96fcd9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682024033812 2023-04-20 20:53:56,355 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f2414b1f85044dc185c9e2c5d7fd40e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682024033900 2023-04-20 20:53:56,355 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0a8584d1d6e547d0876608ed6db253d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682024033900 2023-04-20 20:53:56,355 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d96d9939504c4e4197a427dfb5d8a316, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682024034132 2023-04-20 20:53:56,356 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 41d0c8b4589e4000bd41a78cc8ad3d44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682024034132 2023-04-20 20:53:56,356 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 154754891409478e9c933a352f958ca4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024036053 2023-04-20 20:53:56,360 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 169db961cd9d4c0ebf88bfdfef2812bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024036053 2023-04-20 20:53:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:56,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:53:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:56,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:56,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,365 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:56,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:56,383 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:56,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:56,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:56,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:56,383 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:56,391 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:56,395 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#61 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:56,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024096397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024096400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024096403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,418 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,418 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:56,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:56,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:56,421 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/e4ca2d7d99424d7db253ea9679e17599 2023-04-20 20:53:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:56,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/e68239466243465595ce59042dd66638 2023-04-20 20:53:56,448 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/ebc88d7572874ad7b10fcd212a844316 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/ebc88d7572874ad7b10fcd212a844316 2023-04-20 20:53:56,455 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into ebc88d7572874ad7b10fcd212a844316(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:56,455 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:56,455 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=11, startTime=1682024036348; duration=0sec 2023-04-20 20:53:56,455 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:53:56,455 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:53:56,457 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:53:56,457 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/C is initiating minor compaction (all files) 2023-04-20 20:53:56,457 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/C in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:56,457 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/f8a2f94e0e4e49b4b0a5050c502338f9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/893353891b104f0e9873478e06297be8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/62c49e92df684b0ebeba9a0b3f02c8ea, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/dd7918b719504002806890c227ba680d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/1eefe1566f914e2aadbdae23166cb091] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=60.0 K 2023-04-20 20:53:56,458 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f8a2f94e0e4e49b4b0a5050c502338f9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682024033090 2023-04-20 20:53:56,459 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 893353891b104f0e9873478e06297be8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682024033812 2023-04-20 20:53:56,459 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 62c49e92df684b0ebeba9a0b3f02c8ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682024033900 2023-04-20 20:53:56,460 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting dd7918b719504002806890c227ba680d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682024034132 2023-04-20 20:53:56,460 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1eefe1566f914e2aadbdae23166cb091, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024036053 2023-04-20 20:53:56,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:56,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:56,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:56,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:56,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:56,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:56,478 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,478 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,478 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,478 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:56,478 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:56,478 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,485 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#C#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:53:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024096505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024096505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024096508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,579 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:56,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:56,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:56,612 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:56,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:56,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:56,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:56,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024096626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,629 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:53:56,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:56,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:56,648 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,648 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,648 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:56,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:56,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:56,649 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:56,650 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:56,650 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:56,650 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:56,650 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:56,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,672 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:56,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e1b8bf[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:56,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:56,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:56,697 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,697 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:56,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024096710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024096712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:56,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024096715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:56,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:56,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:56,726 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:56,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:56,730 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:56,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:56,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:56,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:56,730 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:56,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:56,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,782 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,782 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:56,791 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:56,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:56,829 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:56,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:56,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:56,830 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:56,830 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:56,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:56,843 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/4c2c1255dd9a4c37bb74b0f4511f64aa as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/4c2c1255dd9a4c37bb74b0f4511f64aa 2023-04-20 20:53:56,860 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into 4c2c1255dd9a4c37bb74b0f4511f64aa(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:56,860 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:56,860 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=11, startTime=1682024036347; duration=0sec 2023-04-20 20:53:56,860 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:56,887 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/02d45c5c216a4181b862fdb6e9f6e5f7 2023-04-20 20:53:56,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/e4ca2d7d99424d7db253ea9679e17599 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/e4ca2d7d99424d7db253ea9679e17599 2023-04-20 20:53:56,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/e4ca2d7d99424d7db253ea9679e17599, entries=250, sequenceid=453, filesize=16.7 K 2023-04-20 20:53:56,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/e68239466243465595ce59042dd66638 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e68239466243465595ce59042dd66638 2023-04-20 20:53:56,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e68239466243465595ce59042dd66638, entries=150, sequenceid=453, filesize=11.9 K 2023-04-20 20:53:56,930 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:56,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/02d45c5c216a4181b862fdb6e9f6e5f7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/02d45c5c216a4181b862fdb6e9f6e5f7 2023-04-20 20:53:56,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:56,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:56,952 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/9005184fab144faaa009013d05664416 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9005184fab144faaa009013d05664416 2023-04-20 20:53:56,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/02d45c5c216a4181b862fdb6e9f6e5f7, entries=150, sequenceid=453, filesize=11.9 K 2023-04-20 20:53:56,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:56,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:56,954 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:56,954 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:56,955 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:56,955 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:56,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 061610bf32a86961a5dcdce078e5ddc0 in 591ms, sequenceid=453, compaction requested=false 2023-04-20 20:53:56,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:56,955 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,955 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,963 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/C of 061610bf32a86961a5dcdce078e5ddc0 into 9005184fab144faaa009013d05664416(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:53:56,963 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:56,963 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/C, priority=11, startTime=1682024036348; duration=0sec 2023-04-20 20:53:56,963 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:53:56,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:56,987 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,990 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:56,990 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:56,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:56,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:56,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:56,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:56,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:56,995 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:56,995 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:56,996 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:53:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:57,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:57,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:57,036 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/81f09f1387134abc9aed54849104c37b 2023-04-20 20:53:57,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024097046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024097048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024097053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024097155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024097156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024097158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024097359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024097360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024097365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,493 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/69f242870bcd441583f9b6bcf8e48776 2023-04-20 20:53:57,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024097663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024097664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:57,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024097668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,923 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/87cbca502dca4687be4d3b1beaf590ba 2023-04-20 20:53:57,931 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/81f09f1387134abc9aed54849104c37b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/81f09f1387134abc9aed54849104c37b 2023-04-20 20:53:57,938 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/81f09f1387134abc9aed54849104c37b, entries=150, sequenceid=473, filesize=11.9 K 2023-04-20 20:53:57,939 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/69f242870bcd441583f9b6bcf8e48776 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69f242870bcd441583f9b6bcf8e48776 2023-04-20 20:53:57,946 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69f242870bcd441583f9b6bcf8e48776, entries=150, sequenceid=473, filesize=11.9 K 2023-04-20 20:53:57,947 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/87cbca502dca4687be4d3b1beaf590ba as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/87cbca502dca4687be4d3b1beaf590ba 2023-04-20 20:53:57,954 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/87cbca502dca4687be4d3b1beaf590ba, entries=150, sequenceid=473, filesize=11.9 K 2023-04-20 20:53:57,955 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 061610bf32a86961a5dcdce078e5ddc0 in 960ms, sequenceid=473, compaction requested=true 2023-04-20 20:53:57,955 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:57,955 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:57,955 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:57,955 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:57,955 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,958 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:57,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:57,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298e17f6[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:57,960 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:57,960 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,962 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:57,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:57,964 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:57,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:57,967 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:57,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:57,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:57,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:57,967 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:57,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:57,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:57,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:57,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:57,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,972 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,977 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:57,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:57,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:57,978 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:58,078 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:58,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:58,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:58,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:58,082 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:58,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:58,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:58,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:58,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:58,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:58,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:58,086 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:58,086 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:58,086 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:53:58,086 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:58,087 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:58,087 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:58,087 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:58,087 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:58,087 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:58,099 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/213f01594caa4b209934c599e7432a6a 2023-04-20 20:53:58,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:58,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:58,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024098192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024098193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024098194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024098291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,294 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:53:58,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024098296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024098297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024098301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024098501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024098502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024098506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,514 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/09604abd1fe049ba8b7bf80355869bb1 2023-04-20 20:53:58,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024098805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024098806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024098809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,935 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/5f093f76f9ba466685818f6455a8bc2b 2023-04-20 20:53:58,942 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/213f01594caa4b209934c599e7432a6a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/213f01594caa4b209934c599e7432a6a 2023-04-20 20:53:58,949 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/213f01594caa4b209934c599e7432a6a, entries=150, sequenceid=488, filesize=11.9 K 2023-04-20 20:53:58,950 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/09604abd1fe049ba8b7bf80355869bb1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/09604abd1fe049ba8b7bf80355869bb1 2023-04-20 20:53:58,959 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/09604abd1fe049ba8b7bf80355869bb1, entries=150, sequenceid=488, filesize=11.9 K 2023-04-20 20:53:58,960 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/5f093f76f9ba466685818f6455a8bc2b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5f093f76f9ba466685818f6455a8bc2b 2023-04-20 20:53:58,968 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5f093f76f9ba466685818f6455a8bc2b, entries=150, sequenceid=488, filesize=11.9 K 2023-04-20 20:53:58,969 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 061610bf32a86961a5dcdce078e5ddc0 in 883ms, sequenceid=488, compaction requested=true 2023-04-20 20:53:58,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:58,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:58,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:58,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:58,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:58,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:58,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:58,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:58,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:58,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:58,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6e040d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:58,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:58,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:58,985 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:58,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:53:58,986 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,986 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:53:58,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:53:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:53:58,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:53:58,994 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:53:58,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:53:58,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:53:58,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:53:58,994 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:53:58,996 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:58,996 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,996 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:58,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:58,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:58,997 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,000 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,003 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:59,003 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:53:59,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:59,003 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:53:59,003 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:53:59,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:53:59,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:53:59,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:53:59,003 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:59,004 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,004 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:53:59,004 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,004 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:53:59,004 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:53:59,004 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:59,104 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:53:59,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:53:59,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:53:59,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:53:59,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:53:59,108 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:53:59,108 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:59,108 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:53:59,109 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:53:59,109 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:59,109 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:59,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,114 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:53:59,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:53:59,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:59,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:53:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:53:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:53:59,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:53:59,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:53:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:53:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:53:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:53:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:53:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:53:59,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:59,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:53:59,124 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:53:59,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:53:59,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:53:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:53:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:53:59,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:53:59,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:53:59,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024099341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024099342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024099345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024099447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024099447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024099452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,534 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/217c80a797ef468a850508ffd949748f 2023-04-20 20:53:59,548 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/d54c2aec15894ee2a51c27a1fa768adb 2023-04-20 20:53:59,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024099651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024099652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024099661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024099953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024099959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,963 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/b8830de494bc4cc79ccace7357a384f4 2023-04-20 20:53:59,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:53:59,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024099964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,970 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/217c80a797ef468a850508ffd949748f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/217c80a797ef468a850508ffd949748f 2023-04-20 20:53:59,975 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/217c80a797ef468a850508ffd949748f, entries=150, sequenceid=505, filesize=11.9 K 2023-04-20 20:53:59,976 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/d54c2aec15894ee2a51c27a1fa768adb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d54c2aec15894ee2a51c27a1fa768adb 2023-04-20 20:53:59,982 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d54c2aec15894ee2a51c27a1fa768adb, entries=150, sequenceid=505, filesize=11.9 K 2023-04-20 20:53:59,982 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/b8830de494bc4cc79ccace7357a384f4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b8830de494bc4cc79ccace7357a384f4 2023-04-20 20:53:59,987 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b8830de494bc4cc79ccace7357a384f4, entries=150, sequenceid=505, filesize=11.9 K 2023-04-20 20:53:59,988 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 061610bf32a86961a5dcdce078e5ddc0 in 864ms, sequenceid=505, compaction requested=true 2023-04-20 20:53:59,988 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:53:59,988 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:53:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:53:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:53:59,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:53:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:53:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:53:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:53:59,992 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:53:59,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ad3ffbd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:53:59,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:53:59,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,994 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:53:59,994 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:53:59,994 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:53:59,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:00,000 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,000 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:00,001 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:00,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:00,003 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:00,003 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:00,003 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:00,003 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:00,003 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:00,006 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,006 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,006 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,008 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:00,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:00,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:00,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,011 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:00,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:00,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:00,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:00,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:00,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:00,116 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:00,116 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,116 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:00,116 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:00,116 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,117 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,118 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:00,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:00,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:00,125 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:54:00,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:00,126 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:00,126 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:00,126 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:00,126 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:00,126 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:00,166 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/c2d1f0b15ce1472484a5cb8f6c599007 2023-04-20 20:54:00,222 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/008196d0014a49da9e7fdb9eb93ed141 2023-04-20 20:54:00,260 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/76b99547cbff426891f076dd8e54f803 2023-04-20 20:54:00,266 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/c2d1f0b15ce1472484a5cb8f6c599007 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c2d1f0b15ce1472484a5cb8f6c599007 2023-04-20 20:54:00,279 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c2d1f0b15ce1472484a5cb8f6c599007, entries=150, sequenceid=520, filesize=11.9 K 2023-04-20 20:54:00,280 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/008196d0014a49da9e7fdb9eb93ed141 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/008196d0014a49da9e7fdb9eb93ed141 2023-04-20 20:54:00,290 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/008196d0014a49da9e7fdb9eb93ed141, entries=150, sequenceid=520, filesize=11.9 K 2023-04-20 20:54:00,291 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/76b99547cbff426891f076dd8e54f803 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/76b99547cbff426891f076dd8e54f803 2023-04-20 20:54:00,312 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/76b99547cbff426891f076dd8e54f803, entries=150, sequenceid=520, filesize=11.9 K 2023-04-20 20:54:00,313 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 061610bf32a86961a5dcdce078e5ddc0 in 188ms, sequenceid=520, compaction requested=true 2023-04-20 20:54:00,313 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:00,313 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:00,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521f8c18[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:00,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:00,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,322 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:00,324 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:00,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:00,325 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:00,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:00,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:00,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:00,325 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:00,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,328 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,331 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,332 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:00,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:00,335 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:00,336 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:00,436 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:00,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:00,440 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:00,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:00,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:00,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,445 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:00,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:00,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:00,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:00,447 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,447 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:00,448 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:00,448 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:00,449 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,449 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:00,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b77100a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:00,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:00,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,453 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:00,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:00,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:00,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:00,468 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:00,468 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:00,468 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:00,468 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:00,468 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:00,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:00,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:00,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:00,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:00,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:00,471 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:00,471 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:00,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:00,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,481 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:00,481 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:00,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:00,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682024040481, max timestamp=1682024040491Num uniques 859; 2023-04-20 20:54:00,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 20:54:00,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682024040481, max timestamp=1682024040492Num uniques 859; 2023-04-20 20:54:00,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 20:54:00,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682024040481, max timestamp=1682024040492Num uniques 859; 2023-04-20 20:54:00,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 20:54:00,507 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/a45e2c63162f4cd59ca15cec0d05b23b 2023-04-20 20:54:00,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024100520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024100525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024100527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,581 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:00,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:00,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:00,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:00,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:00,591 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:00,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:00,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:00,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,593 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,594 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:00,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:00,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:00,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:00,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:00,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:00,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:00,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:00,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:00,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70423305[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:00,606 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:00,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:00,610 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:00,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:00,616 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:00,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:00,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:00,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:00,617 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:00,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:00,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:00,625 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,625 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,625 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:00,625 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:00,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:00,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024100628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024100635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682024100636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024100636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,646 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:00,725 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:00,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:00,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:00,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:00,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:00,729 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:00,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:00,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:00,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:00,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:00,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:00,732 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,732 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:00,732 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:00,733 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:00,734 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@554904a7[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:00,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:00,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:00,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:00,740 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:00,740 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,740 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:00,743 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:00,743 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:00,743 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:00,743 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:00,743 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:00,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:00,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:00,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,753 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:00,753 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024100832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,853 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:00,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024100845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024100846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:00,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:00,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:00,860 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:00,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:00,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:00,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:00,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:00,867 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,867 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:00,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:00,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:00,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:00,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:00,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:00,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:00,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b983d5a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:00,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:00,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,876 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:00,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:00,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:00,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:00,877 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:00,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:00,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:00,880 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:00,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:00,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:00,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:00,880 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:00,883 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:00,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:00,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,884 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:00,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:00,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:00,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:00,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:00,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:00,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,891 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:00,891 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:00,890 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:00,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:00,891 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:00,891 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:00,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:00,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:00,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:00,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/5355772d965a4c2d9b82a8356f2bf914 2023-04-20 20:54:00,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/a84fae133f76414e83c49b06e1c5c38b 2023-04-20 20:54:00,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/a45e2c63162f4cd59ca15cec0d05b23b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a45e2c63162f4cd59ca15cec0d05b23b 2023-04-20 20:54:00,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a45e2c63162f4cd59ca15cec0d05b23b, entries=100, sequenceid=532, filesize=9.5 K 2023-04-20 20:54:00,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/5355772d965a4c2d9b82a8356f2bf914 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/5355772d965a4c2d9b82a8356f2bf914 2023-04-20 20:54:00,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/5355772d965a4c2d9b82a8356f2bf914, entries=100, sequenceid=532, filesize=9.5 K 2023-04-20 20:54:00,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/a84fae133f76414e83c49b06e1c5c38b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a84fae133f76414e83c49b06e1c5c38b 2023-04-20 20:54:00,991 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:00,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:00,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:01,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:01,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:01,004 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:01,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:01,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:01,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:01,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:01,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a84fae133f76414e83c49b06e1c5c38b, entries=100, sequenceid=532, filesize=9.5 K 2023-04-20 20:54:01,006 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 061610bf32a86961a5dcdce078e5ddc0 in 537ms, sequenceid=532, compaction requested=true 2023-04-20 20:54:01,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:01,006 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 20:54:01,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 20:54:01,007 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 20:54:01,007 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:01,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:01,008 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,008 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:01,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:01,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:01,012 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88420 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 20:54:01,012 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83540 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 20:54:01,012 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:54:01,012 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:54:01,012 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:01,012 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:01,012 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/4c2c1255dd9a4c37bb74b0f4511f64aa, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/e4ca2d7d99424d7db253ea9679e17599, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/81f09f1387134abc9aed54849104c37b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/213f01594caa4b209934c599e7432a6a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/217c80a797ef468a850508ffd949748f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c2d1f0b15ce1472484a5cb8f6c599007, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a45e2c63162f4cd59ca15cec0d05b23b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=86.3 K 2023-04-20 20:54:01,012 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/ebc88d7572874ad7b10fcd212a844316, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e68239466243465595ce59042dd66638, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69f242870bcd441583f9b6bcf8e48776, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/09604abd1fe049ba8b7bf80355869bb1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d54c2aec15894ee2a51c27a1fa768adb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/008196d0014a49da9e7fdb9eb93ed141, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/5355772d965a4c2d9b82a8356f2bf914] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=81.6 K 2023-04-20 20:54:01,013 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4c2c1255dd9a4c37bb74b0f4511f64aa, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024036053 2023-04-20 20:54:01,013 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ebc88d7572874ad7b10fcd212a844316, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024036053 2023-04-20 20:54:01,013 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e4ca2d7d99424d7db253ea9679e17599, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682024036240 2023-04-20 20:54:01,013 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e68239466243465595ce59042dd66638, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682024036240 2023-04-20 20:54:01,014 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 81f09f1387134abc9aed54849104c37b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682024036394 2023-04-20 20:54:01,014 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 69f242870bcd441583f9b6bcf8e48776, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682024036394 2023-04-20 20:54:01,014 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 213f01594caa4b209934c599e7432a6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682024037019 2023-04-20 20:54:01,014 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 09604abd1fe049ba8b7bf80355869bb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682024037019 2023-04-20 20:54:01,014 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 217c80a797ef468a850508ffd949748f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682024038188 2023-04-20 20:54:01,015 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d54c2aec15894ee2a51c27a1fa768adb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682024038188 2023-04-20 20:54:01,015 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c2d1f0b15ce1472484a5cb8f6c599007, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682024039324 2023-04-20 20:54:01,015 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 008196d0014a49da9e7fdb9eb93ed141, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682024039324 2023-04-20 20:54:01,015 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a45e2c63162f4cd59ca15cec0d05b23b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682024040468 2023-04-20 20:54:01,015 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5355772d965a4c2d9b82a8356f2bf914, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682024040468 2023-04-20 20:54:01,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:01,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:01,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:01,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:01,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:01,022 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-20 20:54:01,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:01,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:01,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:01,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:01,023 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:01,023 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:01,044 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#81 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:01,054 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:01,097 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/16d5effb9a354a59bb5d18168f873b14 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/16d5effb9a354a59bb5d18168f873b14 2023-04-20 20:54:01,100 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/aafda76037f24103a0c6ed4e8bd3708b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/aafda76037f24103a0c6ed4e8bd3708b 2023-04-20 20:54:01,112 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into 16d5effb9a354a59bb5d18168f873b14(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:01,112 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:01,112 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=9, startTime=1682024041006; duration=0sec 2023-04-20 20:54:01,113 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:01,113 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 20:54:01,113 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into aafda76037f24103a0c6ed4e8bd3708b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:01,113 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:01,113 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=9, startTime=1682024041006; duration=0sec 2023-04-20 20:54:01,113 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:01,116 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83506 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 20:54:01,116 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/C is initiating minor compaction (all files) 2023-04-20 20:54:01,116 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/C in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:01,116 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9005184fab144faaa009013d05664416, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/02d45c5c216a4181b862fdb6e9f6e5f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/87cbca502dca4687be4d3b1beaf590ba, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5f093f76f9ba466685818f6455a8bc2b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b8830de494bc4cc79ccace7357a384f4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/76b99547cbff426891f076dd8e54f803, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a84fae133f76414e83c49b06e1c5c38b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=81.5 K 2023-04-20 20:54:01,117 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9005184fab144faaa009013d05664416, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024036053 2023-04-20 20:54:01,117 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 02d45c5c216a4181b862fdb6e9f6e5f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682024036240 2023-04-20 20:54:01,119 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 87cbca502dca4687be4d3b1beaf590ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1682024036394 2023-04-20 20:54:01,120 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5f093f76f9ba466685818f6455a8bc2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682024037019 2023-04-20 20:54:01,120 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b8830de494bc4cc79ccace7357a384f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1682024038188 2023-04-20 20:54:01,120 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 76b99547cbff426891f076dd8e54f803, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682024039324 2023-04-20 20:54:01,121 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a84fae133f76414e83c49b06e1c5c38b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682024040468 2023-04-20 20:54:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:01,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:01,141 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#C#compaction#83 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:01,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024101161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024101163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024101164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,172 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/0110ad26cbbe458c986fe0caec3c972d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/0110ad26cbbe458c986fe0caec3c972d 2023-04-20 20:54:01,179 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/C of 061610bf32a86961a5dcdce078e5ddc0 into 0110ad26cbbe458c986fe0caec3c972d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:01,179 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:01,180 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/C, priority=9, startTime=1682024041006; duration=0sec 2023-04-20 20:54:01,180 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:01,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024101267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024101267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024101269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,462 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/0e3184f79fc94e239311371aee092c37 2023-04-20 20:54:01,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024101471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024101476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024101477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,488 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/6ccb97c5cc854126b2f2e619fe7c772c 2023-04-20 20:54:01,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024101774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024101779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:01,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024101780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,908 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/4ed201b81ce943c88e63dce12be4f7dc 2023-04-20 20:54:01,914 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/0e3184f79fc94e239311371aee092c37 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0e3184f79fc94e239311371aee092c37 2023-04-20 20:54:01,921 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0e3184f79fc94e239311371aee092c37, entries=150, sequenceid=565, filesize=11.9 K 2023-04-20 20:54:01,924 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/6ccb97c5cc854126b2f2e619fe7c772c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/6ccb97c5cc854126b2f2e619fe7c772c 2023-04-20 20:54:01,936 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/6ccb97c5cc854126b2f2e619fe7c772c, entries=150, sequenceid=565, filesize=11.9 K 2023-04-20 20:54:01,937 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/4ed201b81ce943c88e63dce12be4f7dc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/4ed201b81ce943c88e63dce12be4f7dc 2023-04-20 20:54:01,944 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/4ed201b81ce943c88e63dce12be4f7dc, entries=150, sequenceid=565, filesize=11.9 K 2023-04-20 20:54:01,944 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 061610bf32a86961a5dcdce078e5ddc0 in 922ms, sequenceid=565, compaction requested=false 2023-04-20 20:54:01,945 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:01,945 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:01,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:01,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:01,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:01,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:01,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:01,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:01,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48dc9563[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:01,951 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:01,951 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,952 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:01,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:01,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:01,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:01,954 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:01,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:01,957 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:01,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:01,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:01,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:01,957 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:01,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:01,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:01,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:01,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:01,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:01,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:01,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:01,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:01,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:01,967 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:01,967 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:01,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:01,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:02,067 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:02,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:02,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:02,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:02,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:02,071 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:02,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:02,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:02,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:02,072 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:02,072 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:02,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:02,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:02,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:02,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:02,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:02,075 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:02,075 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:02,075 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 20:54:02,075 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:02,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:02,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:02,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:02,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:02,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:02,086 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/299909480f944bf293f178ec299f3fc1 2023-04-20 20:54:02,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:02,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:02,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024042297, max timestamp=1682024042308Num uniques 859; 2023-04-20 20:54:02,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:54:02,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024042297, max timestamp=1682024042308Num uniques 859; 2023-04-20 20:54:02,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:54:02,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024042297, max timestamp=1682024042308Num uniques 859; 2023-04-20 20:54:02,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:54:02,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024102337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024102340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024102341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024102341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024102442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024102442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024102446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024102446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,499 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/8a96f7e445334df7baf3a9a94a0fdefb 2023-04-20 20:54:02,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024102645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024102646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024102650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024102650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,917 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/a097009821a847598c9f1dc626b29c7a 2023-04-20 20:54:02,924 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/299909480f944bf293f178ec299f3fc1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/299909480f944bf293f178ec299f3fc1 2023-04-20 20:54:02,937 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/299909480f944bf293f178ec299f3fc1, entries=150, sequenceid=579, filesize=11.9 K 2023-04-20 20:54:02,940 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/8a96f7e445334df7baf3a9a94a0fdefb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8a96f7e445334df7baf3a9a94a0fdefb 2023-04-20 20:54:02,946 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8a96f7e445334df7baf3a9a94a0fdefb, entries=150, sequenceid=579, filesize=11.9 K 2023-04-20 20:54:02,947 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/a097009821a847598c9f1dc626b29c7a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a097009821a847598c9f1dc626b29c7a 2023-04-20 20:54:02,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024102950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024102951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,955 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a097009821a847598c9f1dc626b29c7a, entries=150, sequenceid=579, filesize=11.9 K 2023-04-20 20:54:02,957 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 061610bf32a86961a5dcdce078e5ddc0 in 882ms, sequenceid=579, compaction requested=true 2023-04-20 20:54:02,957 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:02,957 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:02,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:02,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:02,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-20 20:54:02,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:02,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:02,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:02,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:02,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:02,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:02,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:02,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:02,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024102973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024102974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:02,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/6bd3035f49434776bd604c573bacbcb9 2023-04-20 20:54:03,012 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:03,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:03,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:03,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29675d66[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:03,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:03,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,074 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:03,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:03,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024103085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024103085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,112 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:03,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:03,115 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:03,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:03,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:03,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:03,115 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:03,145 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:03,145 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,145 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:03,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,147 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:03,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:03,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:03,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:03,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:03,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,175 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,175 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,175 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:03,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:03,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:03,176 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:03,177 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:03,277 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:03,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:03,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:03,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:03,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:03,282 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:03,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:03,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:03,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:03,283 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,283 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024103287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024103287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:03,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:03,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:03,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:03,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:03,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:03,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:03,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:03,347 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:03,347 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:03,403 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/0fa3fe122ee04f549482dfb1a8af32f7 2023-04-20 20:54:03,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,405 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:03,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:03,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341608d1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:03,411 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:03,411 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,445 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:03,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:03,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:03,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:03,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024103455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024103462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:03,505 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:03,508 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:03,508 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:03,508 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:03,508 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:03,508 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:03,536 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,536 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,536 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:03,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:03,536 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024103593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:03,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024103594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,720 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:03,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:03,720 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:03,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,721 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:03,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:03,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:03,721 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:03,721 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:03,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:03,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:03,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/fc277b84b1384d558238c535f8e646f1 2023-04-20 20:54:03,821 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:03,824 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/6bd3035f49434776bd604c573bacbcb9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6bd3035f49434776bd604c573bacbcb9 2023-04-20 20:54:03,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:03,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:03,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:03,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:03,828 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:03,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:03,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:03,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:03,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,829 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6bd3035f49434776bd604c573bacbcb9, entries=200, sequenceid=614, filesize=14.3 K 2023-04-20 20:54:03,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/0fa3fe122ee04f549482dfb1a8af32f7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/0fa3fe122ee04f549482dfb1a8af32f7 2023-04-20 20:54:03,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/0fa3fe122ee04f549482dfb1a8af32f7, entries=150, sequenceid=614, filesize=11.9 K 2023-04-20 20:54:03,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/fc277b84b1384d558238c535f8e646f1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/fc277b84b1384d558238c535f8e646f1 2023-04-20 20:54:03,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/fc277b84b1384d558238c535f8e646f1, entries=150, sequenceid=614, filesize=11.9 K 2023-04-20 20:54:03,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 061610bf32a86961a5dcdce078e5ddc0 in 888ms, sequenceid=614, compaction requested=true 2023-04-20 20:54:03,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:03,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:03,846 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:03,846 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:03,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:03,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:03,848 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52104 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:03,848 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:54:03,848 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:03,848 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:03,848 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:54:03,848 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/aafda76037f24103a0c6ed4e8bd3708b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0e3184f79fc94e239311371aee092c37, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/299909480f944bf293f178ec299f3fc1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6bd3035f49434776bd604c573bacbcb9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=50.9 K 2023-04-20 20:54:03,848 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:03,849 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/16d5effb9a354a59bb5d18168f873b14, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/6ccb97c5cc854126b2f2e619fe7c772c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8a96f7e445334df7baf3a9a94a0fdefb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/0fa3fe122ee04f549482dfb1a8af32f7] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=48.5 K 2023-04-20 20:54:03,849 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting aafda76037f24103a0c6ed4e8bd3708b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682024039324 2023-04-20 20:54:03,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 16d5effb9a354a59bb5d18168f873b14, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682024039324 2023-04-20 20:54:03,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 6ccb97c5cc854126b2f2e619fe7c772c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1682024040515 2023-04-20 20:54:03,849 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0e3184f79fc94e239311371aee092c37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1682024040515 2023-04-20 20:54:03,850 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8a96f7e445334df7baf3a9a94a0fdefb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1682024041147 2023-04-20 20:54:03,850 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 299909480f944bf293f178ec299f3fc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1682024041147 2023-04-20 20:54:03,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0fa3fe122ee04f549482dfb1a8af32f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1682024042338 2023-04-20 20:54:03,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,851 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6bd3035f49434776bd604c573bacbcb9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1682024042336 2023-04-20 20:54:03,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:03,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:03,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:03,852 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:03,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:03,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:03,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:03,855 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:03,856 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:03,856 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:03,856 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:03,856 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:03,856 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:03,856 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:03,857 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:03,857 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:03,857 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:03,857 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:03,857 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:03,866 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#93 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:03,870 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#94 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:03,885 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/ce4574493a9a4391bf3647d161f6154d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ce4574493a9a4391bf3647d161f6154d 2023-04-20 20:54:03,893 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into ce4574493a9a4391bf3647d161f6154d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:03,894 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:03,894 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=12, startTime=1682024043846; duration=0sec 2023-04-20 20:54:03,894 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:03,894 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:03,895 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:03,896 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/C is initiating minor compaction (all files) 2023-04-20 20:54:03,896 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/C in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:03,896 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/0110ad26cbbe458c986fe0caec3c972d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/4ed201b81ce943c88e63dce12be4f7dc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a097009821a847598c9f1dc626b29c7a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/fc277b84b1384d558238c535f8e646f1] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=48.5 K 2023-04-20 20:54:03,896 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0110ad26cbbe458c986fe0caec3c972d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682024039324 2023-04-20 20:54:03,897 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 4ed201b81ce943c88e63dce12be4f7dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1682024040515 2023-04-20 20:54:03,897 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a097009821a847598c9f1dc626b29c7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1682024041147 2023-04-20 20:54:03,898 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting fc277b84b1384d558238c535f8e646f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1682024042338 2023-04-20 20:54:03,906 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#C#compaction#95 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:03,920 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/13a8e0a706d141349dfb774d06964dfc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/13a8e0a706d141349dfb774d06964dfc 2023-04-20 20:54:03,927 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/C of 061610bf32a86961a5dcdce078e5ddc0 into 13a8e0a706d141349dfb774d06964dfc(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:03,927 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:03,927 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/C, priority=12, startTime=1682024043847; duration=0sec 2023-04-20 20:54:03,927 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:04,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:04,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:04,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024044147, max timestamp=1682024044159Num uniques 859; 2023-04-20 20:54:04,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024044147, max timestamp=1682024044159Num uniques 859; 2023-04-20 20:54:04,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:04,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:04,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024044147, max timestamp=1682024044159Num uniques 859; 2023-04-20 20:54:04,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:04,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024104196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024104204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,275 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/9c8e8fa343e44955a9917741bdf696e2 2023-04-20 20:54:04,285 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/7c25e7aed9e141c88cd8879d74d8c139 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/7c25e7aed9e141c88cd8879d74d8c139 2023-04-20 20:54:04,293 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into 7c25e7aed9e141c88cd8879d74d8c139(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:04,293 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:04,293 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=12, startTime=1682024043846; duration=0sec 2023-04-20 20:54:04,293 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:04,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024104304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024104310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43044 deadline: 1682024104465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43074 deadline: 1682024104468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024104511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024104515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,698 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/e71bb6ecaf53461abcdc9ef651b35eac 2023-04-20 20:54:04,713 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/63db21752ae742ecbbef1f524e4fc568 2023-04-20 20:54:04,719 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/9c8e8fa343e44955a9917741bdf696e2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9c8e8fa343e44955a9917741bdf696e2 2023-04-20 20:54:04,726 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9c8e8fa343e44955a9917741bdf696e2, entries=150, sequenceid=624, filesize=11.9 K 2023-04-20 20:54:04,727 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/e71bb6ecaf53461abcdc9ef651b35eac as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e71bb6ecaf53461abcdc9ef651b35eac 2023-04-20 20:54:04,732 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e71bb6ecaf53461abcdc9ef651b35eac, entries=150, sequenceid=624, filesize=11.9 K 2023-04-20 20:54:04,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/63db21752ae742ecbbef1f524e4fc568 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/63db21752ae742ecbbef1f524e4fc568 2023-04-20 20:54:04,741 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/63db21752ae742ecbbef1f524e4fc568, entries=150, sequenceid=624, filesize=11.9 K 2023-04-20 20:54:04,742 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 061610bf32a86961a5dcdce078e5ddc0 in 886ms, sequenceid=624, compaction requested=false 2023-04-20 20:54:04,743 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:04,743 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:04,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:04,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:04,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:04,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:04,746 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:04,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:04,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:04,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e5e91e4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:04,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,749 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:04,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:04,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:04,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:04,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:04,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:04,754 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:04,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:04,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:04,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:04,754 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:04,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:04,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:04,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:04,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:04,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:04,769 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:04,769 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:04,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:04,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:04,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:04,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-20 20:54:04,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:04,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:04,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:04,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:04,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:04,822 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:04,822 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:04,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024104840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024104841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,869 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:04,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:04,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:04,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:04,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:04,877 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:04,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:04,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:04,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:04,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:04,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:04,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:04,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:04,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:04,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:04,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:04,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:04,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:04,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:04,893 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:04,893 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. started... 2023-04-20 20:54:04,893 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. as already flushing 2023-04-20 20:54:04,893 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:04,895 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:04,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:04,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22856ac1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:04,896 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:04,897 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,899 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:04,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:04,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:04,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:04,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:04,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:04,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:04,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:04,905 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:04,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:04,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:04,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:04,905 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:04,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:04,908 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:04,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,908 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:04,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:04,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:04,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:04,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:04,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:04,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:04,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:04,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:04,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:04,912 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:04,912 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:04,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:04,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:04,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:04,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024104943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:04,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024104947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:04,975 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:62055 2023-04-20 20:54:04,975 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:04,979 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:62055 2023-04-20 20:54:04,979 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:04,980 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:62055 2023-04-20 20:54:04,980 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:04,986 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:62055 2023-04-20 20:54:04,986 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:05,012 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:05,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:05,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:05,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024105148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:05,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:05,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024105154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:05,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/a172f6bcbf4f44aca19cd8444163a18c 2023-04-20 20:54:05,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:05,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024105451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:05,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:05,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024105457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:05,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/e186ad308d0946989b9db51fc21d8829 2023-04-20 20:54:05,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:05,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43092 deadline: 1682024105953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:05,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:05,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43078 deadline: 1682024105963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:06,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/5e82ea35f1244c66839a12f64a069400 2023-04-20 20:54:06,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/a172f6bcbf4f44aca19cd8444163a18c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a172f6bcbf4f44aca19cd8444163a18c 2023-04-20 20:54:06,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a172f6bcbf4f44aca19cd8444163a18c, entries=200, sequenceid=663, filesize=14.3 K 2023-04-20 20:54:06,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/e186ad308d0946989b9db51fc21d8829 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e186ad308d0946989b9db51fc21d8829 2023-04-20 20:54:06,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e186ad308d0946989b9db51fc21d8829, entries=150, sequenceid=663, filesize=11.9 K 2023-04-20 20:54:06,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/5e82ea35f1244c66839a12f64a069400 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5e82ea35f1244c66839a12f64a069400 2023-04-20 20:54:06,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5e82ea35f1244c66839a12f64a069400, entries=150, sequenceid=663, filesize=11.9 K 2023-04-20 20:54:06,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 061610bf32a86961a5dcdce078e5ddc0 in 1287ms, sequenceid=663, compaction requested=true 2023-04-20 20:54:06,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:06,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:06,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:06,108 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:54:06,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:06,108 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:54:06,109 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:54:06,109 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:54:06,109 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/A is initiating minor compaction (all files) 2023-04-20 20:54:06,109 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/B is initiating minor compaction (all files) 2023-04-20 20:54:06,109 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/A in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:06,109 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/B in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:06,109 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ce4574493a9a4391bf3647d161f6154d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9c8e8fa343e44955a9917741bdf696e2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a172f6bcbf4f44aca19cd8444163a18c] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=39.1 K 2023-04-20 20:54:06,109 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/7c25e7aed9e141c88cd8879d74d8c139, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e71bb6ecaf53461abcdc9ef651b35eac, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e186ad308d0946989b9db51fc21d8829] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=36.7 K 2023-04-20 20:54:06,110 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7c25e7aed9e141c88cd8879d74d8c139, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1682024042338 2023-04-20 20:54:06,110 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ce4574493a9a4391bf3647d161f6154d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1682024042338 2023-04-20 20:54:06,110 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e71bb6ecaf53461abcdc9ef651b35eac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682024042968 2023-04-20 20:54:06,110 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9c8e8fa343e44955a9917741bdf696e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682024042968 2023-04-20 20:54:06,111 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e186ad308d0946989b9db51fc21d8829, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682024044194 2023-04-20 20:54:06,111 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a172f6bcbf4f44aca19cd8444163a18c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682024044194 2023-04-20 20:54:06,117 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#B#compaction#101 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:06,117 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#A#compaction#102 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:06,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:06,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:06,483 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:62055 2023-04-20 20:54:06,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:06,483 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:06,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:06,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:06,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:06,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:06,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:06,491 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:62055 2023-04-20 20:54:06,491 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:06,528 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/83416932e76e426a9831a6ff8849a8ae as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/83416932e76e426a9831a6ff8849a8ae 2023-04-20 20:54:06,529 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/c4750a48c70c4c018133980c3ffe2365 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c4750a48c70c4c018133980c3ffe2365 2023-04-20 20:54:06,534 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/B of 061610bf32a86961a5dcdce078e5ddc0 into 83416932e76e426a9831a6ff8849a8ae(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:06,534 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:06,534 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/B, priority=13, startTime=1682024046108; duration=0sec 2023-04-20 20:54:06,534 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:06,534 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:54:06,534 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/A of 061610bf32a86961a5dcdce078e5ddc0 into c4750a48c70c4c018133980c3ffe2365(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:06,534 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:06,534 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/A, priority=13, startTime=1682024046108; duration=0sec 2023-04-20 20:54:06,534 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:06,535 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:54:06,535 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 061610bf32a86961a5dcdce078e5ddc0/C is initiating minor compaction (all files) 2023-04-20 20:54:06,535 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 061610bf32a86961a5dcdce078e5ddc0/C in TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:06,535 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/13a8e0a706d141349dfb774d06964dfc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/63db21752ae742ecbbef1f524e4fc568, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5e82ea35f1244c66839a12f64a069400] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp, totalSize=36.7 K 2023-04-20 20:54:06,536 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 13a8e0a706d141349dfb774d06964dfc, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1682024042338 2023-04-20 20:54:06,536 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 63db21752ae742ecbbef1f524e4fc568, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682024042968 2023-04-20 20:54:06,537 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5e82ea35f1244c66839a12f64a069400, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682024044194 2023-04-20 20:54:06,543 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 061610bf32a86961a5dcdce078e5ddc0#C#compaction#104 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:06,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/2cc62335c046455ca03f35fa04a10d9c 2023-04-20 20:54:06,953 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/d4998601057d497f869599e8594ad3a2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/d4998601057d497f869599e8594ad3a2 2023-04-20 20:54:06,959 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 061610bf32a86961a5dcdce078e5ddc0/C of 061610bf32a86961a5dcdce078e5ddc0 into d4998601057d497f869599e8594ad3a2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:06,959 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:06,959 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0., storeName=061610bf32a86961a5dcdce078e5ddc0/C, priority=13, startTime=1682024046108; duration=0sec 2023-04-20 20:54:06,959 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:06,962 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:62055 2023-04-20 20:54:06,962 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:06,970 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:62055 2023-04-20 20:54:06,971 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:07,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/f39bbe19fb2c4a89aa6583b29f8dd7a1 2023-04-20 20:54:07,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/6c5284fc4fee4ecc87083f05ba2bc665 2023-04-20 20:54:07,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/2cc62335c046455ca03f35fa04a10d9c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2cc62335c046455ca03f35fa04a10d9c 2023-04-20 20:54:07,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2cc62335c046455ca03f35fa04a10d9c, entries=150, sequenceid=673, filesize=11.9 K 2023-04-20 20:54:07,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/f39bbe19fb2c4a89aa6583b29f8dd7a1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f39bbe19fb2c4a89aa6583b29f8dd7a1 2023-04-20 20:54:07,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f39bbe19fb2c4a89aa6583b29f8dd7a1, entries=150, sequenceid=673, filesize=11.9 K 2023-04-20 20:54:07,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/6c5284fc4fee4ecc87083f05ba2bc665 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/6c5284fc4fee4ecc87083f05ba2bc665 2023-04-20 20:54:07,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/6c5284fc4fee4ecc87083f05ba2bc665, entries=150, sequenceid=673, filesize=11.9 K 2023-04-20 20:54:07,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=20.13 KB/20610 for 061610bf32a86961a5dcdce078e5ddc0 in 1262ms, sequenceid=673, compaction requested=false 2023-04-20 20:54:07,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:10,744 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:62055 2023-04-20 20:54:10,744 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:10,744 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 163 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 140 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 3702 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 3639 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1625 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 4870 rows 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1611 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 4831 rows 2023-04-20 20:54:10,745 INFO [Listener at localhost/45619] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 20:54:10,745 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:62055 2023-04-20 20:54:10,745 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:10,750 INFO [Listener at localhost/45619] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 20:54:10,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-20 20:54:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:10,765 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024050765"}]},"ts":"1682024050765"} 2023-04-20 20:54:10,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 20:54:10,766 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 20:54:10,850 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 20:54:10,852 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=061610bf32a86961a5dcdce078e5ddc0, UNASSIGN}] 2023-04-20 20:54:10,853 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=061610bf32a86961a5dcdce078e5ddc0, UNASSIGN 2023-04-20 20:54:10,854 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=061610bf32a86961a5dcdce078e5ddc0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:10,854 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024050854"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024050854"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024050854"}]},"ts":"1682024050854"} 2023-04-20 20:54:10,855 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:54:10,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 20:54:11,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:11,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 061610bf32a86961a5dcdce078e5ddc0, disabling compactions & flushes 2023-04-20 20:54:11,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:11,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:11,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. after waiting 0 ms 2023-04-20 20:54:11,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:11,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 061610bf32a86961a5dcdce078e5ddc0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 20:54:11,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=A 2023-04-20 20:54:11,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:11,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=B 2023-04-20 20:54:11,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:11,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 061610bf32a86961a5dcdce078e5ddc0, store=C 2023-04-20 20:54:11,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:11,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 20:54:11,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 20:54:11,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/09fc583f7df945529ec92e3215432aee 2023-04-20 20:54:11,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/054cf294e1d54196b2c242ce38c47b28 2023-04-20 20:54:11,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 20:54:12,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/242c0a5b4bb84d208caa589067fac610 2023-04-20 20:54:12,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/A/09fc583f7df945529ec92e3215432aee as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/09fc583f7df945529ec92e3215432aee 2023-04-20 20:54:12,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/09fc583f7df945529ec92e3215432aee, entries=150, sequenceid=683, filesize=11.9 K 2023-04-20 20:54:12,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/B/054cf294e1d54196b2c242ce38c47b28 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/054cf294e1d54196b2c242ce38c47b28 2023-04-20 20:54:12,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/054cf294e1d54196b2c242ce38c47b28, entries=150, sequenceid=683, filesize=11.9 K 2023-04-20 20:54:12,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/.tmp/C/242c0a5b4bb84d208caa589067fac610 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/242c0a5b4bb84d208caa589067fac610 2023-04-20 20:54:12,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/242c0a5b4bb84d208caa589067fac610, entries=150, sequenceid=683, filesize=11.9 K 2023-04-20 20:54:12,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 061610bf32a86961a5dcdce078e5ddc0 in 1255ms, sequenceid=683, compaction requested=true 2023-04-20 20:54:12,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cfd4805581b84b02a462a41e7a79effb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/879fea6a840c4842b01b307b5d8c1399, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/26161f06a2fd425993093e8e231b2f25, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/51203484ad2744228ce3ede64579e080, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9f8211c7e55c4d579a47715f1cd91c96, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/008ae815e4c34c73a0ec9e4601de4284, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ed53622e1a4f4c6dacbb34a15b7b77cb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/bf5840d21d68484f8733e839ed61ae44, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cf8034b45e434df7b7f008f296e83e3c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a688a4a7dfcd4538a22576f071cfa27c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/da8130d7df964651bb834856ca8d3eb7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/34cae5e9034045479e43ea6bcaceceac, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9dc342cda4e04d748b6dc50656b2548e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/84fd3287138541049b217293d4ceed3f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6ff7b6cfd354487989dbaec517794b1d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2201b1a8f5d24251b1ac1c822f50496d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/754664cb4ce644a08e08ff28ad96fcd9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0a8584d1d6e547d0876608ed6db253d5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/41d0c8b4589e4000bd41a78cc8ad3d44, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/4c2c1255dd9a4c37bb74b0f4511f64aa, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/169db961cd9d4c0ebf88bfdfef2812bd, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/e4ca2d7d99424d7db253ea9679e17599, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/81f09f1387134abc9aed54849104c37b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/213f01594caa4b209934c599e7432a6a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/217c80a797ef468a850508ffd949748f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c2d1f0b15ce1472484a5cb8f6c599007, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/aafda76037f24103a0c6ed4e8bd3708b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a45e2c63162f4cd59ca15cec0d05b23b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0e3184f79fc94e239311371aee092c37, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/299909480f944bf293f178ec299f3fc1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6bd3035f49434776bd604c573bacbcb9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ce4574493a9a4391bf3647d161f6154d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9c8e8fa343e44955a9917741bdf696e2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a172f6bcbf4f44aca19cd8444163a18c] to archive 2023-04-20 20:54:12,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:54:12,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cfd4805581b84b02a462a41e7a79effb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cfd4805581b84b02a462a41e7a79effb 2023-04-20 20:54:12,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/879fea6a840c4842b01b307b5d8c1399 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/879fea6a840c4842b01b307b5d8c1399 2023-04-20 20:54:12,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/26161f06a2fd425993093e8e231b2f25 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/26161f06a2fd425993093e8e231b2f25 2023-04-20 20:54:12,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/51203484ad2744228ce3ede64579e080 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/51203484ad2744228ce3ede64579e080 2023-04-20 20:54:12,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9f8211c7e55c4d579a47715f1cd91c96 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9f8211c7e55c4d579a47715f1cd91c96 2023-04-20 20:54:12,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/008ae815e4c34c73a0ec9e4601de4284 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/008ae815e4c34c73a0ec9e4601de4284 2023-04-20 20:54:12,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ed53622e1a4f4c6dacbb34a15b7b77cb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ed53622e1a4f4c6dacbb34a15b7b77cb 2023-04-20 20:54:12,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/bf5840d21d68484f8733e839ed61ae44 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/bf5840d21d68484f8733e839ed61ae44 2023-04-20 20:54:12,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cf8034b45e434df7b7f008f296e83e3c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/cf8034b45e434df7b7f008f296e83e3c 2023-04-20 20:54:12,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a688a4a7dfcd4538a22576f071cfa27c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a688a4a7dfcd4538a22576f071cfa27c 2023-04-20 20:54:12,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/da8130d7df964651bb834856ca8d3eb7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/da8130d7df964651bb834856ca8d3eb7 2023-04-20 20:54:12,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/34cae5e9034045479e43ea6bcaceceac to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/34cae5e9034045479e43ea6bcaceceac 2023-04-20 20:54:12,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9dc342cda4e04d748b6dc50656b2548e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9dc342cda4e04d748b6dc50656b2548e 2023-04-20 20:54:12,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/84fd3287138541049b217293d4ceed3f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/84fd3287138541049b217293d4ceed3f 2023-04-20 20:54:12,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6ff7b6cfd354487989dbaec517794b1d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6ff7b6cfd354487989dbaec517794b1d 2023-04-20 20:54:12,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2201b1a8f5d24251b1ac1c822f50496d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2201b1a8f5d24251b1ac1c822f50496d 2023-04-20 20:54:12,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/754664cb4ce644a08e08ff28ad96fcd9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/754664cb4ce644a08e08ff28ad96fcd9 2023-04-20 20:54:12,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0a8584d1d6e547d0876608ed6db253d5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0a8584d1d6e547d0876608ed6db253d5 2023-04-20 20:54:12,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/41d0c8b4589e4000bd41a78cc8ad3d44 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/41d0c8b4589e4000bd41a78cc8ad3d44 2023-04-20 20:54:12,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/4c2c1255dd9a4c37bb74b0f4511f64aa to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/4c2c1255dd9a4c37bb74b0f4511f64aa 2023-04-20 20:54:12,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/169db961cd9d4c0ebf88bfdfef2812bd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/169db961cd9d4c0ebf88bfdfef2812bd 2023-04-20 20:54:12,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/e4ca2d7d99424d7db253ea9679e17599 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/e4ca2d7d99424d7db253ea9679e17599 2023-04-20 20:54:12,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/81f09f1387134abc9aed54849104c37b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/81f09f1387134abc9aed54849104c37b 2023-04-20 20:54:12,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/213f01594caa4b209934c599e7432a6a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/213f01594caa4b209934c599e7432a6a 2023-04-20 20:54:12,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/217c80a797ef468a850508ffd949748f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/217c80a797ef468a850508ffd949748f 2023-04-20 20:54:12,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c2d1f0b15ce1472484a5cb8f6c599007 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c2d1f0b15ce1472484a5cb8f6c599007 2023-04-20 20:54:12,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/aafda76037f24103a0c6ed4e8bd3708b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/aafda76037f24103a0c6ed4e8bd3708b 2023-04-20 20:54:12,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a45e2c63162f4cd59ca15cec0d05b23b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a45e2c63162f4cd59ca15cec0d05b23b 2023-04-20 20:54:12,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0e3184f79fc94e239311371aee092c37 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/0e3184f79fc94e239311371aee092c37 2023-04-20 20:54:12,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/299909480f944bf293f178ec299f3fc1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/299909480f944bf293f178ec299f3fc1 2023-04-20 20:54:12,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6bd3035f49434776bd604c573bacbcb9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/6bd3035f49434776bd604c573bacbcb9 2023-04-20 20:54:12,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ce4574493a9a4391bf3647d161f6154d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/ce4574493a9a4391bf3647d161f6154d 2023-04-20 20:54:12,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9c8e8fa343e44955a9917741bdf696e2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/9c8e8fa343e44955a9917741bdf696e2 2023-04-20 20:54:12,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a172f6bcbf4f44aca19cd8444163a18c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/a172f6bcbf4f44aca19cd8444163a18c 2023-04-20 20:54:12,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/90cbb8a2bdb5432487b5b0cafad4c746, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/39fcf30ca3254658a3fbadb7a843d537, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/860498852b4a4d1daf416132c35b91ed, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b423963d10ca41909475d2c70d6546f4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8daf3a4bb65d42f3a0af9ebff3a461e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b2337bc9e278403b9a9a84f9859a7ba0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/45ff9f7dc24a4dff9a6eace3ac1cff33, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/85d14189d1fc426186e66b7f40436881, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69a52f6ff4714ddda20fbfc2b16f8ba3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/27d37e1c38b34f859a8f5ab54a6cf7bb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/31d158c884f64242810c190c4c59e460, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/28cd84d83b874aafb3f1d85e260874ce, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d691c9edb86349189fa577999b9443ec, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/cc3b1c84540a4c8fa8d230f855a15e4b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/57f784f84a6e4b728d4a47158025390a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/1453848fa21e49a892bbbc2f6da7ef1a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b26c34664c1049b196be5403d6560d97, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f2414b1f85044dc185c9e2c5d7fd40e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d96d9939504c4e4197a427dfb5d8a316, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/ebc88d7572874ad7b10fcd212a844316, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/154754891409478e9c933a352f958ca4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e68239466243465595ce59042dd66638, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69f242870bcd441583f9b6bcf8e48776, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/09604abd1fe049ba8b7bf80355869bb1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d54c2aec15894ee2a51c27a1fa768adb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/008196d0014a49da9e7fdb9eb93ed141, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/16d5effb9a354a59bb5d18168f873b14, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/5355772d965a4c2d9b82a8356f2bf914, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/6ccb97c5cc854126b2f2e619fe7c772c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8a96f7e445334df7baf3a9a94a0fdefb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/7c25e7aed9e141c88cd8879d74d8c139, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/0fa3fe122ee04f549482dfb1a8af32f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e71bb6ecaf53461abcdc9ef651b35eac, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e186ad308d0946989b9db51fc21d8829] to archive 2023-04-20 20:54:12,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:54:12,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/90cbb8a2bdb5432487b5b0cafad4c746 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/90cbb8a2bdb5432487b5b0cafad4c746 2023-04-20 20:54:12,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/39fcf30ca3254658a3fbadb7a843d537 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/39fcf30ca3254658a3fbadb7a843d537 2023-04-20 20:54:12,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/860498852b4a4d1daf416132c35b91ed to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/860498852b4a4d1daf416132c35b91ed 2023-04-20 20:54:12,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b423963d10ca41909475d2c70d6546f4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b423963d10ca41909475d2c70d6546f4 2023-04-20 20:54:12,372 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8daf3a4bb65d42f3a0af9ebff3a461e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8daf3a4bb65d42f3a0af9ebff3a461e7 2023-04-20 20:54:12,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b2337bc9e278403b9a9a84f9859a7ba0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b2337bc9e278403b9a9a84f9859a7ba0 2023-04-20 20:54:12,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/45ff9f7dc24a4dff9a6eace3ac1cff33 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/45ff9f7dc24a4dff9a6eace3ac1cff33 2023-04-20 20:54:12,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/85d14189d1fc426186e66b7f40436881 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/85d14189d1fc426186e66b7f40436881 2023-04-20 20:54:12,376 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69a52f6ff4714ddda20fbfc2b16f8ba3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69a52f6ff4714ddda20fbfc2b16f8ba3 2023-04-20 20:54:12,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/27d37e1c38b34f859a8f5ab54a6cf7bb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/27d37e1c38b34f859a8f5ab54a6cf7bb 2023-04-20 20:54:12,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/31d158c884f64242810c190c4c59e460 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/31d158c884f64242810c190c4c59e460 2023-04-20 20:54:12,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/28cd84d83b874aafb3f1d85e260874ce to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/28cd84d83b874aafb3f1d85e260874ce 2023-04-20 20:54:12,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d691c9edb86349189fa577999b9443ec to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d691c9edb86349189fa577999b9443ec 2023-04-20 20:54:12,381 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/cc3b1c84540a4c8fa8d230f855a15e4b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/cc3b1c84540a4c8fa8d230f855a15e4b 2023-04-20 20:54:12,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/57f784f84a6e4b728d4a47158025390a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/57f784f84a6e4b728d4a47158025390a 2023-04-20 20:54:12,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/1453848fa21e49a892bbbc2f6da7ef1a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/1453848fa21e49a892bbbc2f6da7ef1a 2023-04-20 20:54:12,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b26c34664c1049b196be5403d6560d97 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/b26c34664c1049b196be5403d6560d97 2023-04-20 20:54:12,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f2414b1f85044dc185c9e2c5d7fd40e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f2414b1f85044dc185c9e2c5d7fd40e7 2023-04-20 20:54:12,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d96d9939504c4e4197a427dfb5d8a316 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d96d9939504c4e4197a427dfb5d8a316 2023-04-20 20:54:12,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/ebc88d7572874ad7b10fcd212a844316 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/ebc88d7572874ad7b10fcd212a844316 2023-04-20 20:54:12,389 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/154754891409478e9c933a352f958ca4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/154754891409478e9c933a352f958ca4 2023-04-20 20:54:12,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e68239466243465595ce59042dd66638 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e68239466243465595ce59042dd66638 2023-04-20 20:54:12,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69f242870bcd441583f9b6bcf8e48776 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/69f242870bcd441583f9b6bcf8e48776 2023-04-20 20:54:12,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/09604abd1fe049ba8b7bf80355869bb1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/09604abd1fe049ba8b7bf80355869bb1 2023-04-20 20:54:12,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d54c2aec15894ee2a51c27a1fa768adb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/d54c2aec15894ee2a51c27a1fa768adb 2023-04-20 20:54:12,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/008196d0014a49da9e7fdb9eb93ed141 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/008196d0014a49da9e7fdb9eb93ed141 2023-04-20 20:54:12,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/16d5effb9a354a59bb5d18168f873b14 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/16d5effb9a354a59bb5d18168f873b14 2023-04-20 20:54:12,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/5355772d965a4c2d9b82a8356f2bf914 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/5355772d965a4c2d9b82a8356f2bf914 2023-04-20 20:54:12,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/6ccb97c5cc854126b2f2e619fe7c772c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/6ccb97c5cc854126b2f2e619fe7c772c 2023-04-20 20:54:12,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8a96f7e445334df7baf3a9a94a0fdefb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/8a96f7e445334df7baf3a9a94a0fdefb 2023-04-20 20:54:12,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/7c25e7aed9e141c88cd8879d74d8c139 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/7c25e7aed9e141c88cd8879d74d8c139 2023-04-20 20:54:12,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/0fa3fe122ee04f549482dfb1a8af32f7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/0fa3fe122ee04f549482dfb1a8af32f7 2023-04-20 20:54:12,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e71bb6ecaf53461abcdc9ef651b35eac to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e71bb6ecaf53461abcdc9ef651b35eac 2023-04-20 20:54:12,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e186ad308d0946989b9db51fc21d8829 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/e186ad308d0946989b9db51fc21d8829 2023-04-20 20:54:12,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/2881c715e8d849eaa2ea029e1f702a12, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/e8eb93b351774831aae63a941e646e81, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/43393ff1d80149e487a8e5144e3c1459, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/05d978102e134910a17dbe5a9ae492f8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/490ff071b22e4832a45423146aa8cb96, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9cf1f6d5b4aa477d977ea070d3aa97e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b3ec2075d0c643799d998cdce54a1d4b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/23a107b66d8f4a0b871c61c1af9ede42, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/8ded717aa5a14384a8b0a11d6440a652, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/052fe2f9a0b94d089136c6740f687fe0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/69f4740c9cd94e2bab42f34474aff4ff, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/824712ac404b41b7a56ffecbcaca12cd, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/06a4527abf8940a9a938c2a6a5d1cdac, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/f8a2f94e0e4e49b4b0a5050c502338f9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/993aa8898eb648bab6182da846059bca, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/893353891b104f0e9873478e06297be8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/62c49e92df684b0ebeba9a0b3f02c8ea, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/dd7918b719504002806890c227ba680d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9005184fab144faaa009013d05664416, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/1eefe1566f914e2aadbdae23166cb091, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/02d45c5c216a4181b862fdb6e9f6e5f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/87cbca502dca4687be4d3b1beaf590ba, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5f093f76f9ba466685818f6455a8bc2b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b8830de494bc4cc79ccace7357a384f4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/76b99547cbff426891f076dd8e54f803, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/0110ad26cbbe458c986fe0caec3c972d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a84fae133f76414e83c49b06e1c5c38b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/4ed201b81ce943c88e63dce12be4f7dc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a097009821a847598c9f1dc626b29c7a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/13a8e0a706d141349dfb774d06964dfc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/fc277b84b1384d558238c535f8e646f1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/63db21752ae742ecbbef1f524e4fc568, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5e82ea35f1244c66839a12f64a069400] to archive 2023-04-20 20:54:12,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:54:12,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/2881c715e8d849eaa2ea029e1f702a12 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/2881c715e8d849eaa2ea029e1f702a12 2023-04-20 20:54:12,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/e8eb93b351774831aae63a941e646e81 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/e8eb93b351774831aae63a941e646e81 2023-04-20 20:54:12,415 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/43393ff1d80149e487a8e5144e3c1459 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/43393ff1d80149e487a8e5144e3c1459 2023-04-20 20:54:12,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/05d978102e134910a17dbe5a9ae492f8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/05d978102e134910a17dbe5a9ae492f8 2023-04-20 20:54:12,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/490ff071b22e4832a45423146aa8cb96 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/490ff071b22e4832a45423146aa8cb96 2023-04-20 20:54:12,419 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9cf1f6d5b4aa477d977ea070d3aa97e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9cf1f6d5b4aa477d977ea070d3aa97e7 2023-04-20 20:54:12,421 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b3ec2075d0c643799d998cdce54a1d4b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b3ec2075d0c643799d998cdce54a1d4b 2023-04-20 20:54:12,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/23a107b66d8f4a0b871c61c1af9ede42 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/23a107b66d8f4a0b871c61c1af9ede42 2023-04-20 20:54:12,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/8ded717aa5a14384a8b0a11d6440a652 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/8ded717aa5a14384a8b0a11d6440a652 2023-04-20 20:54:12,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/052fe2f9a0b94d089136c6740f687fe0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/052fe2f9a0b94d089136c6740f687fe0 2023-04-20 20:54:12,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/69f4740c9cd94e2bab42f34474aff4ff to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/69f4740c9cd94e2bab42f34474aff4ff 2023-04-20 20:54:12,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/824712ac404b41b7a56ffecbcaca12cd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/824712ac404b41b7a56ffecbcaca12cd 2023-04-20 20:54:12,429 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/06a4527abf8940a9a938c2a6a5d1cdac to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/06a4527abf8940a9a938c2a6a5d1cdac 2023-04-20 20:54:12,430 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/f8a2f94e0e4e49b4b0a5050c502338f9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/f8a2f94e0e4e49b4b0a5050c502338f9 2023-04-20 20:54:12,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/993aa8898eb648bab6182da846059bca to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/993aa8898eb648bab6182da846059bca 2023-04-20 20:54:12,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/893353891b104f0e9873478e06297be8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/893353891b104f0e9873478e06297be8 2023-04-20 20:54:12,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/62c49e92df684b0ebeba9a0b3f02c8ea to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/62c49e92df684b0ebeba9a0b3f02c8ea 2023-04-20 20:54:12,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/dd7918b719504002806890c227ba680d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/dd7918b719504002806890c227ba680d 2023-04-20 20:54:12,436 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9005184fab144faaa009013d05664416 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/9005184fab144faaa009013d05664416 2023-04-20 20:54:12,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/1eefe1566f914e2aadbdae23166cb091 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/1eefe1566f914e2aadbdae23166cb091 2023-04-20 20:54:12,438 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/02d45c5c216a4181b862fdb6e9f6e5f7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/02d45c5c216a4181b862fdb6e9f6e5f7 2023-04-20 20:54:12,439 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/87cbca502dca4687be4d3b1beaf590ba to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/87cbca502dca4687be4d3b1beaf590ba 2023-04-20 20:54:12,441 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5f093f76f9ba466685818f6455a8bc2b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5f093f76f9ba466685818f6455a8bc2b 2023-04-20 20:54:12,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b8830de494bc4cc79ccace7357a384f4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/b8830de494bc4cc79ccace7357a384f4 2023-04-20 20:54:12,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/76b99547cbff426891f076dd8e54f803 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/76b99547cbff426891f076dd8e54f803 2023-04-20 20:54:12,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/0110ad26cbbe458c986fe0caec3c972d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/0110ad26cbbe458c986fe0caec3c972d 2023-04-20 20:54:12,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a84fae133f76414e83c49b06e1c5c38b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a84fae133f76414e83c49b06e1c5c38b 2023-04-20 20:54:12,447 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/4ed201b81ce943c88e63dce12be4f7dc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/4ed201b81ce943c88e63dce12be4f7dc 2023-04-20 20:54:12,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a097009821a847598c9f1dc626b29c7a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/a097009821a847598c9f1dc626b29c7a 2023-04-20 20:54:12,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/13a8e0a706d141349dfb774d06964dfc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/13a8e0a706d141349dfb774d06964dfc 2023-04-20 20:54:12,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/fc277b84b1384d558238c535f8e646f1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/fc277b84b1384d558238c535f8e646f1 2023-04-20 20:54:12,467 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/63db21752ae742ecbbef1f524e4fc568 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/63db21752ae742ecbbef1f524e4fc568 2023-04-20 20:54:12,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5e82ea35f1244c66839a12f64a069400 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/5e82ea35f1244c66839a12f64a069400 2023-04-20 20:54:12,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/recovered.edits/686.seqid, newMaxSeqId=686, maxSeqId=1 2023-04-20 20:54:12,476 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0. 2023-04-20 20:54:12,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 061610bf32a86961a5dcdce078e5ddc0: 2023-04-20 20:54:12,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:12,478 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=061610bf32a86961a5dcdce078e5ddc0, regionState=CLOSED 2023-04-20 20:54:12,479 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024052478"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024052478"}]},"ts":"1682024052478"} 2023-04-20 20:54:12,481 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-20 20:54:12,482 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 061610bf32a86961a5dcdce078e5ddc0, server=jenkins-hbase4.apache.org,39223,1682024020002 in 1.6250 sec 2023-04-20 20:54:12,483 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-20 20:54:12,483 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=061610bf32a86961a5dcdce078e5ddc0, UNASSIGN in 1.6290 sec 2023-04-20 20:54:12,484 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024052483"}]},"ts":"1682024052483"} 2023-04-20 20:54:12,484 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 20:54:12,486 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 20:54:12,488 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7300 sec 2023-04-20 20:54:12,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 20:54:12,871 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-20 20:54:12,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-20 20:54:12,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:12,886 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:12,888 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:12,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-20 20:54:12,896 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:12,899 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/recovered.edits] 2023-04-20 20:54:12,903 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/09fc583f7df945529ec92e3215432aee to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/09fc583f7df945529ec92e3215432aee 2023-04-20 20:54:12,905 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2cc62335c046455ca03f35fa04a10d9c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/2cc62335c046455ca03f35fa04a10d9c 2023-04-20 20:54:12,906 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c4750a48c70c4c018133980c3ffe2365 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/A/c4750a48c70c4c018133980c3ffe2365 2023-04-20 20:54:12,909 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/054cf294e1d54196b2c242ce38c47b28 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/054cf294e1d54196b2c242ce38c47b28 2023-04-20 20:54:12,911 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/83416932e76e426a9831a6ff8849a8ae to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/83416932e76e426a9831a6ff8849a8ae 2023-04-20 20:54:12,912 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f39bbe19fb2c4a89aa6583b29f8dd7a1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/B/f39bbe19fb2c4a89aa6583b29f8dd7a1 2023-04-20 20:54:12,915 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/242c0a5b4bb84d208caa589067fac610 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/242c0a5b4bb84d208caa589067fac610 2023-04-20 20:54:12,917 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/6c5284fc4fee4ecc87083f05ba2bc665 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/6c5284fc4fee4ecc87083f05ba2bc665 2023-04-20 20:54:12,920 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/d4998601057d497f869599e8594ad3a2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/C/d4998601057d497f869599e8594ad3a2 2023-04-20 20:54:12,923 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/recovered.edits/686.seqid to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0/recovered.edits/686.seqid 2023-04-20 20:54:12,924 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/061610bf32a86961a5dcdce078e5ddc0 2023-04-20 20:54:12,924 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:54:12,927 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:12,954 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 20:54:13,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-20 20:54:13,006 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 20:54:13,008 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:13,008 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 20:54:13,008 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682024053008"}]},"ts":"9223372036854775807"} 2023-04-20 20:54:13,011 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 20:54:13,011 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 061610bf32a86961a5dcdce078e5ddc0, NAME => 'TestAcidGuarantees,,1682024023667.061610bf32a86961a5dcdce078e5ddc0.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 20:54:13,011 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 20:54:13,011 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682024053011"}]},"ts":"9223372036854775807"} 2023-04-20 20:54:13,019 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 20:54:13,021 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:13,023 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 144 msec 2023-04-20 20:54:13,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-20 20:54:13,206 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-20 20:54:13,223 INFO [Listener at localhost/45619] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=232 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:39223-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bbc8bb2-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bbc8bb2-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-940390530_15 at /127.0.0.1:60114 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bbc8bb2-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bbc8bb2-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=560 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=487 (was 357) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=8851 (was 9594) 2023-04-20 20:54:13,234 INFO [Listener at localhost/45619] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=487, ProcessCount=169, AvailableMemoryMB=8850 2023-04-20 20:54:13,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:54:13,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:54:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:13,240 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 20:54:13,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-20 20:54:13,241 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 20:54:13,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 20:54:13,242 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:13,243 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 empty. 2023-04-20 20:54:13,244 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:13,244 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:54:13,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 20:54:13,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 20:54:13,655 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:54:13,657 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3479312d5cfad95b1d49f86d4ca52095, NAME => 'TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp 2023-04-20 20:54:13,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 20:54:14,064 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:54:14,064 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3479312d5cfad95b1d49f86d4ca52095, disabling compactions & flushes 2023-04-20 20:54:14,064 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,064 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,064 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. after waiting 0 ms 2023-04-20 20:54:14,064 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,064 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,065 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:14,067 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 20:54:14,068 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024054068"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024054068"}]},"ts":"1682024054068"} 2023-04-20 20:54:14,070 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 20:54:14,070 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 20:54:14,071 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024054071"}]},"ts":"1682024054071"} 2023-04-20 20:54:14,072 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 20:54:14,084 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, ASSIGN}] 2023-04-20 20:54:14,086 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, ASSIGN 2023-04-20 20:54:14,087 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=false 2023-04-20 20:54:14,238 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:14,239 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024054238"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024054238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024054238"}]},"ts":"1682024054238"} 2023-04-20 20:54:14,240 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:54:14,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 20:54:14,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3479312d5cfad95b1d49f86d4ca52095, NAME => 'TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:54:14,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:54:14,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,398 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,400 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A 2023-04-20 20:54:14,400 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A 2023-04-20 20:54:14,400 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:14,400 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3479312d5cfad95b1d49f86d4ca52095 columnFamilyName A 2023-04-20 20:54:14,401 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(310): Store=3479312d5cfad95b1d49f86d4ca52095/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:14,401 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,402 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B 2023-04-20 20:54:14,402 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B 2023-04-20 20:54:14,402 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:14,402 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3479312d5cfad95b1d49f86d4ca52095 columnFamilyName B 2023-04-20 20:54:14,403 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(310): Store=3479312d5cfad95b1d49f86d4ca52095/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:14,403 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,404 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C 2023-04-20 20:54:14,404 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C 2023-04-20 20:54:14,404 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:14,404 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3479312d5cfad95b1d49f86d4ca52095 columnFamilyName C 2023-04-20 20:54:14,405 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(310): Store=3479312d5cfad95b1d49f86d4ca52095/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:14,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:54:14,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:14,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:54:14,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3479312d5cfad95b1d49f86d4ca52095; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74629883, jitterRate=0.1120719164609909}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:54:14,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:14,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., pid=18, masterSystemTime=1682024054392 2023-04-20 20:54:14,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:14,414 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:14,415 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024054414"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024054414"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024054414"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024054414"}]},"ts":"1682024054414"} 2023-04-20 20:54:14,418 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-20 20:54:14,418 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 in 176 msec 2023-04-20 20:54:14,419 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-20 20:54:14,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, ASSIGN in 334 msec 2023-04-20 20:54:14,420 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 20:54:14,420 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024054420"}]},"ts":"1682024054420"} 2023-04-20 20:54:14,421 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 20:54:14,423 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 20:54:14,424 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1860 sec 2023-04-20 20:54:15,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 20:54:15,347 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-20 20:54:15,348 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x52274650 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:15,359 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7829b43e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:15,360 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:15,365 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45300, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:15,366 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 20:54:15,368 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 20:54:15,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:54:15,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:54:15,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:15,796 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 20:54:15,798 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:54:15,798 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 20:54:15,802 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-20 20:54:15,815 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, REOPEN/MOVE}] 2023-04-20 20:54:15,816 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, REOPEN/MOVE 2023-04-20 20:54:15,817 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:15,817 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024055817"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024055817"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024055817"}]},"ts":"1682024055817"} 2023-04-20 20:54:15,818 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:54:15,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3479312d5cfad95b1d49f86d4ca52095, disabling compactions & flushes 2023-04-20 20:54:15,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. after waiting 0 ms 2023-04-20 20:54:15,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:15,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-20 20:54:15,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:15,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:15,975 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 3479312d5cfad95b1d49f86d4ca52095 to self. 2023-04-20 20:54:15,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:15,977 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=CLOSED 2023-04-20 20:54:15,978 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024055977"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024055977"}]},"ts":"1682024055977"} 2023-04-20 20:54:15,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-20 20:54:15,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 in 161 msec 2023-04-20 20:54:15,981 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=true 2023-04-20 20:54:16,131 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,131 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024056131"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024056131"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024056131"}]},"ts":"1682024056131"} 2023-04-20 20:54:16,133 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:54:16,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3479312d5cfad95b1d49f86d4ca52095, NAME => 'TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:54:16,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:54:16,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,291 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,292 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A 2023-04-20 20:54:16,292 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A 2023-04-20 20:54:16,292 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:16,295 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3479312d5cfad95b1d49f86d4ca52095 columnFamilyName A 2023-04-20 20:54:16,296 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(310): Store=3479312d5cfad95b1d49f86d4ca52095/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:16,297 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,298 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B 2023-04-20 20:54:16,298 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B 2023-04-20 20:54:16,298 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:16,299 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3479312d5cfad95b1d49f86d4ca52095 columnFamilyName B 2023-04-20 20:54:16,299 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(310): Store=3479312d5cfad95b1d49f86d4ca52095/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:16,299 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,300 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C 2023-04-20 20:54:16,300 DEBUG [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C 2023-04-20 20:54:16,300 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:16,301 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3479312d5cfad95b1d49f86d4ca52095 columnFamilyName C 2023-04-20 20:54:16,301 INFO [StoreOpener-3479312d5cfad95b1d49f86d4ca52095-1] regionserver.HStore(310): Store=3479312d5cfad95b1d49f86d4ca52095/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:16,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:54:16,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3479312d5cfad95b1d49f86d4ca52095; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74136819, jitterRate=0.1047246903181076}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:54:16,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:16,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., pid=23, masterSystemTime=1682024056285 2023-04-20 20:54:16,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,309 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,310 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024056309"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024056309"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024056309"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024056309"}]},"ts":"1682024056309"} 2023-04-20 20:54:16,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-20 20:54:16,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 in 178 msec 2023-04-20 20:54:16,314 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-20 20:54:16,314 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, REOPEN/MOVE in 497 msec 2023-04-20 20:54:16,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-20 20:54:16,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-20 20:54:16,319 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-04-20 20:54:16,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-20 20:54:16,328 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x34135972 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,332 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33a47c05, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,333 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,337 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,337 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,340 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,341 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,347 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,347 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,350 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,351 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,357 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,357 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,361 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,362 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,365 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,365 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:16,368 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:16,377 DEBUG [hconnection-0x46262b6e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,381 DEBUG [hconnection-0x3254ccd3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,381 DEBUG [hconnection-0x5bf8eed2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,381 DEBUG [hconnection-0x39788fe3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,381 DEBUG [hconnection-0x25adaa4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,382 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,382 DEBUG [hconnection-0x2294e7ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,383 DEBUG [hconnection-0xc480d09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,383 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,384 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:16,396 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,396 DEBUG [hconnection-0x1696913d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,396 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,396 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,397 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,398 DEBUG [hconnection-0x1ec56119-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:16,406 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,406 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:16,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:16,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:16,414 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:16,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:16,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:16,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:16,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:54:16,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:16,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:16,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:16,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:16,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:16,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:16,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,417 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,418 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,418 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:16,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:16,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:16,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024116439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024116439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024116440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024116441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024116442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024056433, max timestamp=1682024056439Num uniques 859; 2023-04-20 20:54:16,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:16,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024056433, max timestamp=1682024056439Num uniques 859; 2023-04-20 20:54:16,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:16,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024056433, max timestamp=1682024056439Num uniques 859; 2023-04-20 20:54:16,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:16,447 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,447 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:16,447 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:16,447 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:16,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:16,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:16,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311a9f9a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:16,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:16,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,460 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:16,460 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,460 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:16,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,463 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:16,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:16,465 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:16,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:16,466 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:16,466 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:16,466 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:16,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,476 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,476 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:16,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:16,478 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:16,478 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:16,503 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304202753947fe9614bb6b6b1e74a22afb159 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202753947fe9614bb6b6b1e74a22afb159 2023-04-20 20:54:16,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024116557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024116558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024116557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024116562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024116563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,578 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:16,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:16,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:16,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:16,586 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:16,586 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,586 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:16,586 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:16,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:16,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:16,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:16,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:16,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:16,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:16,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:16,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:16,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:16,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:16,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:16,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:16,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:16,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,606 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:16,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ce59e8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:16,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:16,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,610 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:16,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:16,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:16,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:16,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:16,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:16,618 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:16,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:16,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:16,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:16,618 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:16,620 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,620 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,621 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:16,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:16,627 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,627 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:16,627 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:16,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:16,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024116664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024116664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024116664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024116666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024116667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,727 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:16,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:16,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:16,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:16,734 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:16,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:16,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:16,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:16,740 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:16,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:16,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:16,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:16,744 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,744 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:16,744 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:16,745 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:16,749 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:16,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:16,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe53f04[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:16,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:16,752 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:16,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:16,755 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:16,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:16,758 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:16,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:16,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:16,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:16,758 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:16,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:16,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,768 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:16,768 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:16,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,868 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:16,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:16,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024116868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024116868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024116869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:16,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024116870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:16,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024116870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:16,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:16,873 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:16,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:16,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:16,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,877 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:16,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:16,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:16,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:16,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:16,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:16,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:16,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:16,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:16,882 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:16,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46bf975b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:16,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:16,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,889 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:16,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:16,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:16,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:16,891 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:16,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:16,894 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:16,894 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:16,894 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:16,894 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:16,894 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:16,896 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,896 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,896 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:16,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,897 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:16,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:16,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:16,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:16,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:16,902 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:16,902 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:16,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:16,918 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f903aa554894459f8d49a43ce83bcdfc 2023-04-20 20:54:17,002 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:17,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:17,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:17,006 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:17,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:17,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:17,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,009 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:17,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:17,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:17,011 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:17,011 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,011 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:17,011 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:17,011 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:17,011 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,011 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:17,011 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:17,012 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:17,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:17,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@233c3659[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:17,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:17,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,017 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:17,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:17,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:17,018 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,018 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:17,019 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:17,021 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:17,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:17,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:17,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:17,021 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:17,023 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,023 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,024 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,026 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:17,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:17,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:17,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,028 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:17,028 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:17,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,129 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:17,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:17,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:17,133 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:17,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:17,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:17,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,134 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:17,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:17,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:17,136 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,136 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:17,137 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:17,137 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:17,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:17,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:17,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:17,138 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,138 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,139 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:17,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647a2acf[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:17,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:17,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,142 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:17,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:17,144 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,144 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,144 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,144 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:17,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:17,149 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:17,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:17,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:17,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:17,149 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:17,151 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,151 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,151 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,154 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:17,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:17,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,157 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,157 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,157 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,157 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:17,157 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:17,157 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:17,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024117172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024117173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024117173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024117173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024117174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,257 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:17,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:17,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:17,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:17,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:17,261 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:17,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:17,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:17,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,264 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,264 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:17,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:17,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:17,266 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,266 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:17,266 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:17,266 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:17,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:17,270 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,270 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,271 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@281245b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:17,273 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:17,273 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:17,275 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,275 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:17,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:17,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:17,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:17,281 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:17,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:17,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:17,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:17,281 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:17,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,283 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,286 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,286 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:17,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:17,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,287 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:17,287 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:17,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:17,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/c0584a32196e4c83934603348c6ce1c3 2023-04-20 20:54:17,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/12c11b2282d84252ab11dcc66f739038 2023-04-20 20:54:17,387 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:17,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:17,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:17,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:17,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:17,392 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:17,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f903aa554894459f8d49a43ce83bcdfc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f903aa554894459f8d49a43ce83bcdfc 2023-04-20 20:54:17,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:17,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:17,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,395 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:17,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:17,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:17,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:17,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:17,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f903aa554894459f8d49a43ce83bcdfc, entries=150, sequenceid=16, filesize=25.1 K 2023-04-20 20:54:17,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:17,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:17,399 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,399 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:17,399 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:17,399 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/c0584a32196e4c83934603348c6ce1c3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/c0584a32196e4c83934603348c6ce1c3 2023-04-20 20:54:17,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:17,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:17,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:17,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:17,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/c0584a32196e4c83934603348c6ce1c3, entries=150, sequenceid=16, filesize=11.6 K 2023-04-20 20:54:17,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/12c11b2282d84252ab11dcc66f739038 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/12c11b2282d84252ab11dcc66f739038 2023-04-20 20:54:17,407 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,408 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,408 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:17,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383e7e1f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:17,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:17,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,411 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:17,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:17,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:17,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:17,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:17,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:17,413 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:17,417 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:17,417 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:17,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:17,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:17,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/12c11b2282d84252ab11dcc66f739038, entries=150, sequenceid=16, filesize=11.6 K 2023-04-20 20:54:17,426 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 3479312d5cfad95b1d49f86d4ca52095 in 1010ms, sequenceid=16, compaction requested=false 2023-04-20 20:54:17,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:17,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,430 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:17,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:17,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:17,430 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:17,430 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:17,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:17,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,431 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,431 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,431 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:17,431 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:17,530 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:17,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:17,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:17,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:17,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:17,537 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:17,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:17,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:17,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:17,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:17,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:17,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:17,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:17,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:17,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:17,541 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:17,541 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:17,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:17,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:17,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:17,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:17,544 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 20:54:17,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:17,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:17,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:17,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:17,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:17,545 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:17,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:17,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:17,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024117682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024117684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024117685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024117685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024117686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024117786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024117788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024117788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024117789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024117790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,966 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420a621346717214f6b8a41da30c4fed8cf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a621346717214f6b8a41da30c4fed8cf 2023-04-20 20:54:17,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024117991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024117991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024117992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024117992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:17,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024117995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,038 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 20:54:18,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024118297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024118297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024118297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024118298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024118299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,372 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/4e82ca4c119f4cc99feedd125e7b616f 2023-04-20 20:54:18,388 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/e32e123686504c5da9d11b6b9c72d7e7 2023-04-20 20:54:18,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024118801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024118802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024118802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024118803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:18,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024118804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,816 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/f2b60286cab043df988e0639150c6532 2023-04-20 20:54:18,822 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/4e82ca4c119f4cc99feedd125e7b616f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/4e82ca4c119f4cc99feedd125e7b616f 2023-04-20 20:54:18,827 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/4e82ca4c119f4cc99feedd125e7b616f, entries=150, sequenceid=51, filesize=25.1 K 2023-04-20 20:54:18,828 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/e32e123686504c5da9d11b6b9c72d7e7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e32e123686504c5da9d11b6b9c72d7e7 2023-04-20 20:54:18,833 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e32e123686504c5da9d11b6b9c72d7e7, entries=150, sequenceid=51, filesize=11.6 K 2023-04-20 20:54:18,834 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/f2b60286cab043df988e0639150c6532 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/f2b60286cab043df988e0639150c6532 2023-04-20 20:54:18,838 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/f2b60286cab043df988e0639150c6532, entries=150, sequenceid=51, filesize=11.6 K 2023-04-20 20:54:18,839 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 3479312d5cfad95b1d49f86d4ca52095 in 1295ms, sequenceid=51, compaction requested=false 2023-04-20 20:54:18,839 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:18,839 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:18,839 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:18,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:18,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a7209bf[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:18,843 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:18,843 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,844 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,844 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,844 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:18,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:18,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:18,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:18,848 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:18,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:18,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:18,851 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:18,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:18,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:18,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:18,851 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:18,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:18,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:18,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:18,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:18,855 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:18,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:18,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:18,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:18,857 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:18,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:18,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:18,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:18,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:18,858 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:18,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:18,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:18,858 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:18,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:18,958 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:18,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:18,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:18,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:18,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:18,962 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:18,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:18,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:18,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:18,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:18,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:18,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:18,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:18,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:18,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:18,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:18,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:18,968 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:18,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:18,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:18,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:18,987 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042061e0c31f4982415f878ff00a5d79594e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042061e0c31f4982415f878ff00a5d79594e 2023-04-20 20:54:18,999 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/ed9337049e2c48fb86a1ed785ce4c5bd 2023-04-20 20:54:19,415 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/fde71672592245769e38a0470e6b80c4 2023-04-20 20:54:19,443 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/208159da14754cfe8e04d6a8027eb615 2023-04-20 20:54:19,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/ed9337049e2c48fb86a1ed785ce4c5bd as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ed9337049e2c48fb86a1ed785ce4c5bd 2023-04-20 20:54:19,457 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ed9337049e2c48fb86a1ed785ce4c5bd, entries=150, sequenceid=63, filesize=25.1 K 2023-04-20 20:54:19,458 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/fde71672592245769e38a0470e6b80c4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fde71672592245769e38a0470e6b80c4 2023-04-20 20:54:19,465 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fde71672592245769e38a0470e6b80c4, entries=150, sequenceid=63, filesize=11.6 K 2023-04-20 20:54:19,466 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/208159da14754cfe8e04d6a8027eb615 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/208159da14754cfe8e04d6a8027eb615 2023-04-20 20:54:19,478 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/208159da14754cfe8e04d6a8027eb615, entries=150, sequenceid=63, filesize=11.6 K 2023-04-20 20:54:19,479 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 3479312d5cfad95b1d49f86d4ca52095 in 511ms, sequenceid=63, compaction requested=true 2023-04-20 20:54:19,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:19,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:19,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:19,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4feb686c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:19,484 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:19,484 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,491 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:19,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,496 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:19,500 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:19,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:19,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:19,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:19,500 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:19,501 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,501 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:19,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:19,508 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,508 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,508 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:19,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,509 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,609 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:19,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:19,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:19,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:19,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:19,616 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:19,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:19,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:19,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:19,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:19,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:19,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:19,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:19,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:19,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:19,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:19,624 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:19,624 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:19,624 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:19,624 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:19,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:19,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c33e36[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:19,629 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:19,629 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:19,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:19,633 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,633 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:19,633 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,633 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:19,636 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:19,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:19,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:19,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:19,636 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:19,638 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,638 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,638 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,642 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,645 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:19,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,646 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:19,646 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:19,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:19,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,746 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:19,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:19,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:19,750 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:19,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:19,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:19,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:19,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:19,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:19,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:19,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:19,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:19,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:19,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:19,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:19,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:19,756 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:19,756 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:19,756 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:19,756 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:19,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:19,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:19,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:19,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:19,759 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,759 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,759 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,759 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc66e37[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:19,761 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:19,761 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,762 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:19,762 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:19,762 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:19,763 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,763 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:19,764 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:19,768 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:19,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:19,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:19,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:19,768 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:19,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,772 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,775 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,775 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,775 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:19,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:19,775 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,775 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:19,775 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:19,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:19,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,776 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:19,776 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:19,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:19,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:19,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:19,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:19,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:19,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:19,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:19,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:19,876 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:19,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:19,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:19,896 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:19,896 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:19,896 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:19,896 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:19,897 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,897 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,898 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:19,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:19,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:19,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:19,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:19,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:19,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:19,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:19,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:19,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:19,901 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:19,901 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:19,901 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:19,902 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:19,904 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,905 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420a26ee3a98ee5483699d1061805072be1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a26ee3a98ee5483699d1061805072be1 2023-04-20 20:54:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,906 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,906 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:19,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6593a181[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:19,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:19,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:19,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:19,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682024059892, max timestamp=1682024059910Num uniques 859; 2023-04-20 20:54:19,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:19,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682024059892, max timestamp=1682024059910Num uniques 859; 2023-04-20 20:54:19,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024059892, max timestamp=1682024059910Num uniques 859; 2023-04-20 20:54:19,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-20 20:54:19,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:19,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:19,918 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:19,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:19,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:19,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:19,918 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:19,921 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=75, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/a43965da9d2448e990bc3b7977f1e752 2023-04-20 20:54:19,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:19,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,925 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,934 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:19,934 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:19,934 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:19,934 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:19,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:19,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:19,935 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:19,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:19,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:19,935 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:19,935 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:19,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:19,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024119936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024119938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024119939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024119946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:19,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024119947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,035 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:20,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:20,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:20,040 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:20,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:20,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:20,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:20,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:20,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:20,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:20,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:20,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:20,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:20,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:20,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:20,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:20,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:20,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712f0d57[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:20,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024120048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024120048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024120048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,052 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,052 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:20,053 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:20,054 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,054 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,054 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:20,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024120053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,058 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:20,058 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:20,058 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:20,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024120053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:20,058 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:20,058 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:20,064 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:20,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,070 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,070 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:20,070 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,171 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:20,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:20,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:20,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:20,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:20,174 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:20,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:20,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:20,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,176 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:20,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:20,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:20,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:20,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:20,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:20,180 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cc0bfca[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:20,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:20,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,184 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:20,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:20,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:20,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:20,188 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:20,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:20,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:20,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:20,188 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:20,190 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,190 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,190 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,191 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:20,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,195 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:20,195 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024120253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024120253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024120254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024120259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024120260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,295 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:20,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:20,299 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:20,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:20,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:20,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:20,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:20,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:20,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:20,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:20,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:20,304 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,304 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:20,304 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:20,304 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:20,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,305 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:20,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@104bcbb3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:20,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:20,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:20,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:20,322 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:20,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:20,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:20,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:20,322 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:20,325 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,325 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,325 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,333 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:20,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:20,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:20,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,334 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:20,334 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:20,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/6395b536ddba4ccd99cfd6551552809e 2023-04-20 20:54:20,434 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:20,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:20,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:20,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:20,439 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:20,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:20,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:20,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,442 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,442 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:20,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:20,444 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:20,447 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:20,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@682cef92[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:20,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:20,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:20,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:20,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:20,453 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:20,456 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:20,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:20,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:20,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:20,457 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:20,458 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,458 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,458 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,464 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:20,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:20,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:20,464 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:20,464 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:20,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024120556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024120556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024120557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024120561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,565 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:20,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:20,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:20,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024120564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:20,568 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:20,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:20,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:20,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:20,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:20,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:20,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:20,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:20,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:20,578 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,579 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:20,579 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:20,579 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:20,580 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,580 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,581 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,581 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61601275[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:20,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:20,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,583 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:20,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:20,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:20,584 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,584 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:20,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:20,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,586 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:20,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:20,589 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:20,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:20,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:20,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:20,589 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:20,591 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,591 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,591 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:20,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,595 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:20,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,596 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:20,696 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:20,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:20,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:20,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:20,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:20,699 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:20,699 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,699 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:20,699 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:20,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,701 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:20,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:20,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:20,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:20,704 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,704 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:20,704 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:20,704 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:20,706 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,706 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f7cf7c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:20,708 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:20,709 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:20,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:20,711 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:20,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:20,714 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:20,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:20,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:20,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:20,714 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:20,716 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,716 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:20,716 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:20,716 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:20,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:20,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,722 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,722 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:20,722 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:20,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/ecfe9f645563426e9e0af64223538819 2023-04-20 20:54:20,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/a43965da9d2448e990bc3b7977f1e752 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/a43965da9d2448e990bc3b7977f1e752 2023-04-20 20:54:20,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/a43965da9d2448e990bc3b7977f1e752, entries=350, sequenceid=75, filesize=52.3 K 2023-04-20 20:54:20,789 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/6395b536ddba4ccd99cfd6551552809e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6395b536ddba4ccd99cfd6551552809e 2023-04-20 20:54:20,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6395b536ddba4ccd99cfd6551552809e, entries=150, sequenceid=75, filesize=11.6 K 2023-04-20 20:54:20,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/ecfe9f645563426e9e0af64223538819 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/ecfe9f645563426e9e0af64223538819 2023-04-20 20:54:20,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/ecfe9f645563426e9e0af64223538819, entries=150, sequenceid=75, filesize=11.6 K 2023-04-20 20:54:20,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=142.73 KB/146156 for 3479312d5cfad95b1d49f86d4ca52095 in 924ms, sequenceid=75, compaction requested=true 2023-04-20 20:54:20,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:20,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:20,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:20,800 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:20,800 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:20,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:20,801 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 130748 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:20,801 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/A is initiating minor compaction (all files) 2023-04-20 20:54:20,802 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:20,802 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/A in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,802 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/B is initiating minor compaction (all files) 2023-04-20 20:54:20,802 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f903aa554894459f8d49a43ce83bcdfc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/4e82ca4c119f4cc99feedd125e7b616f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ed9337049e2c48fb86a1ed785ce4c5bd, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/a43965da9d2448e990bc3b7977f1e752] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=127.7 K 2023-04-20 20:54:20,802 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/B in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,802 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/c0584a32196e4c83934603348c6ce1c3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e32e123686504c5da9d11b6b9c72d7e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fde71672592245769e38a0470e6b80c4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6395b536ddba4ccd99cfd6551552809e] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=46.4 K 2023-04-20 20:54:20,802 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f903aa554894459f8d49a43ce83bcdfc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682024056411 2023-04-20 20:54:20,802 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c0584a32196e4c83934603348c6ce1c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682024056411 2023-04-20 20:54:20,803 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 4e82ca4c119f4cc99feedd125e7b616f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682024056555 2023-04-20 20:54:20,803 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e32e123686504c5da9d11b6b9c72d7e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682024056555 2023-04-20 20:54:20,803 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ed9337049e2c48fb86a1ed785ce4c5bd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682024057683 2023-04-20 20:54:20,803 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting fde71672592245769e38a0470e6b80c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682024057683 2023-04-20 20:54:20,804 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a43965da9d2448e990bc3b7977f1e752, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682024059808 2023-04-20 20:54:20,804 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 6395b536ddba4ccd99cfd6551552809e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682024059808 2023-04-20 20:54:20,817 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#B#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:20,822 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:20,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:20,833 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:20,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:20,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:20,834 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,834 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,838 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#A#compaction#123 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:20,839 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:20,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:20,839 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:20,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:20,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:20,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:20,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:20,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:20,843 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/1d460e0c12a84558b152f4235c92a7ae as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/1d460e0c12a84558b152f4235c92a7ae 2023-04-20 20:54:20,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:20,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:20,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:20,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:20,844 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=142.73 KB heapSize=404.38 KB 2023-04-20 20:54:20,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:20,845 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:20,845 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:20,845 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:20,845 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:20,845 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:20,849 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/B of 3479312d5cfad95b1d49f86d4ca52095 into 1d460e0c12a84558b152f4235c92a7ae(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:20,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:20,849 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/B, priority=12, startTime=1682024060800; duration=0sec 2023-04-20 20:54:20,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:20,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:20,850 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:20,850 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/C is initiating minor compaction (all files) 2023-04-20 20:54:20,851 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/C in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:20,851 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/12c11b2282d84252ab11dcc66f739038, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/f2b60286cab043df988e0639150c6532, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/208159da14754cfe8e04d6a8027eb615, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/ecfe9f645563426e9e0af64223538819] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=46.4 K 2023-04-20 20:54:20,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 12c11b2282d84252ab11dcc66f739038, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682024056411 2023-04-20 20:54:20,852 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f2b60286cab043df988e0639150c6532, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682024056555 2023-04-20 20:54:20,852 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 208159da14754cfe8e04d6a8027eb615, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682024057683 2023-04-20 20:54:20,854 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ecfe9f645563426e9e0af64223538819, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682024059808 2023-04-20 20:54:20,870 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#C#compaction#125 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:20,879 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/4e0ea1b371b54d2d9bb8288ae3955bde as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/4e0ea1b371b54d2d9bb8288ae3955bde 2023-04-20 20:54:20,884 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/C of 3479312d5cfad95b1d49f86d4ca52095 into 4e0ea1b371b54d2d9bb8288ae3955bde(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:20,884 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:20,884 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/C, priority=12, startTime=1682024060800; duration=0sec 2023-04-20 20:54:20,884 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:21,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:21,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:21,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024121066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024121067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024121067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024121069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024121069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024121168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024121173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024121173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,270 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304202ce12673a29f4cfb9555ddf460afe9a2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202ce12673a29f4cfb9555ddf460afe9a2 2023-04-20 20:54:21,281 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/312f95734f3743449ec77d878b454124 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/312f95734f3743449ec77d878b454124 2023-04-20 20:54:21,286 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/48f5ff774dc44295bd0ed05342a8be87 2023-04-20 20:54:21,292 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/A of 3479312d5cfad95b1d49f86d4ca52095 into 312f95734f3743449ec77d878b454124(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:21,292 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:21,292 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/A, priority=12, startTime=1682024060800; duration=0sec 2023-04-20 20:54:21,292 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:21,309 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/e08845360406493e8060a0a41db7948a 2023-04-20 20:54:21,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024121371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024121375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024121376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024121676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024121680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:21,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024121680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,740 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/0347f183a1a64de5bfeedcdf4a7af0ad 2023-04-20 20:54:21,745 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/48f5ff774dc44295bd0ed05342a8be87 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/48f5ff774dc44295bd0ed05342a8be87 2023-04-20 20:54:21,750 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/48f5ff774dc44295bd0ed05342a8be87, entries=150, sequenceid=112, filesize=25.1 K 2023-04-20 20:54:21,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/e08845360406493e8060a0a41db7948a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e08845360406493e8060a0a41db7948a 2023-04-20 20:54:21,754 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e08845360406493e8060a0a41db7948a, entries=150, sequenceid=112, filesize=11.6 K 2023-04-20 20:54:21,755 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/0347f183a1a64de5bfeedcdf4a7af0ad as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0347f183a1a64de5bfeedcdf4a7af0ad 2023-04-20 20:54:21,759 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0347f183a1a64de5bfeedcdf4a7af0ad, entries=150, sequenceid=112, filesize=11.6 K 2023-04-20 20:54:21,760 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~142.73 KB/146156, heapSize ~404.34 KB/414040, currentSize=40.25 KB/41220 for 3479312d5cfad95b1d49f86d4ca52095 in 916ms, sequenceid=112, compaction requested=false 2023-04-20 20:54:21,760 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:21,760 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:21,761 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:21,761 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:21,763 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f9382[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:21,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:21,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:21,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:21,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:21,774 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:21,774 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:21,774 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:21,774 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:21,775 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:21,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:21,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,776 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:21,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:21,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:21,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:21,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:21,782 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:21,782 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:21,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:21,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:21,882 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:21,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:21,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:21,886 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:21,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:21,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:21,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:21,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:21,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:21,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:21,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:21,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:21,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:21,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:21,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:21,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:21,890 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:21,890 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:21,890 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-20 20:54:21,891 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:21,891 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:21,891 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:21,891 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:21,891 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:21,891 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:21,906 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304203eb06a41d75c49f7985e988ac16d2694 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203eb06a41d75c49f7985e988ac16d2694 2023-04-20 20:54:21,912 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=124, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/7732968ce84a43b191134e55ea41cd4e 2023-04-20 20:54:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:22,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:22,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024062094, max timestamp=1682024062104Num uniques 859; 2023-04-20 20:54:22,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024062094, max timestamp=1682024062104Num uniques 859; 2023-04-20 20:54:22,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:22,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:22,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024062094, max timestamp=1682024062104Num uniques 859; 2023-04-20 20:54:22,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:22,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024122131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024122132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024122181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024122184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024122185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024122237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024122237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,322 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/4c8b41f812b94e819336d15c395a2c99 2023-04-20 20:54:22,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024122441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024122441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,734 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/45685b5104774c4fb301a817b59d396a 2023-04-20 20:54:22,740 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/7732968ce84a43b191134e55ea41cd4e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/7732968ce84a43b191134e55ea41cd4e 2023-04-20 20:54:22,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024122745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,747 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/7732968ce84a43b191134e55ea41cd4e, entries=150, sequenceid=124, filesize=25.1 K 2023-04-20 20:54:22,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:22,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024122746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/4c8b41f812b94e819336d15c395a2c99 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/4c8b41f812b94e819336d15c395a2c99 2023-04-20 20:54:22,757 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/4c8b41f812b94e819336d15c395a2c99, entries=150, sequenceid=124, filesize=11.6 K 2023-04-20 20:54:22,758 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/45685b5104774c4fb301a817b59d396a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/45685b5104774c4fb301a817b59d396a 2023-04-20 20:54:22,763 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/45685b5104774c4fb301a817b59d396a, entries=150, sequenceid=124, filesize=11.6 K 2023-04-20 20:54:22,764 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 3479312d5cfad95b1d49f86d4ca52095 in 874ms, sequenceid=124, compaction requested=true 2023-04-20 20:54:22,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:22,771 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:22,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:22,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:22,774 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,774 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:22,774 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,774 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@970bcaa[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:22,776 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:22,777 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,778 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:22,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:22,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:22,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:22,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:22,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:22,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:22,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:22,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:22,784 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:22,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:22,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:22,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:22,784 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:22,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:22,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:22,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:22,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:22,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:22,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:22,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:22,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:22,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:22,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:22,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:22,796 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:22,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:22,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:22,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,796 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:22,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:22,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:22,896 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:22,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:22,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:22,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:22,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:22,901 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:22,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:22,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:22,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:22,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:22,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:22,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:22,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:22,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:22,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:22,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:22,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:22,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:22,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:22,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:22,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:22,908 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:22,908 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:22,908 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:22,908 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:22,908 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-20 20:54:22,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:22,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:22,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:22,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:22,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:22,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:23,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:23,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:23,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024123195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024123196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024123197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024123249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024123249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024123300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024123300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,340 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042086078f9f6f3443cf822a91d84b597fdd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042086078f9f6f3443cf822a91d84b597fdd 2023-04-20 20:54:23,345 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=161, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/9859022c51ce4140908e6fb5c59726d5 2023-04-20 20:54:23,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024123501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024123502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,762 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/aefeef38560f44beb30642e509f33acf 2023-04-20 20:54:23,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024123807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:23,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:23,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024123808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,175 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/8e2c3d2e3f7749a18406f418fea92e45 2023-04-20 20:54:24,180 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/9859022c51ce4140908e6fb5c59726d5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9859022c51ce4140908e6fb5c59726d5 2023-04-20 20:54:24,184 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9859022c51ce4140908e6fb5c59726d5, entries=150, sequenceid=161, filesize=25.3 K 2023-04-20 20:54:24,185 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/aefeef38560f44beb30642e509f33acf as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aefeef38560f44beb30642e509f33acf 2023-04-20 20:54:24,189 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aefeef38560f44beb30642e509f33acf, entries=150, sequenceid=161, filesize=11.8 K 2023-04-20 20:54:24,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/8e2c3d2e3f7749a18406f418fea92e45 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8e2c3d2e3f7749a18406f418fea92e45 2023-04-20 20:54:24,194 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8e2c3d2e3f7749a18406f418fea92e45, entries=150, sequenceid=161, filesize=11.8 K 2023-04-20 20:54:24,195 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 3479312d5cfad95b1d49f86d4ca52095 in 1287ms, sequenceid=161, compaction requested=true 2023-04-20 20:54:24,195 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:24,195 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:24,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:24,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:24,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:24,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:24,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e2f44b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:24,199 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:24,199 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:24,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:24,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,203 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:24,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:24,208 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:24,208 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:24,208 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:24,208 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:24,208 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:24,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,210 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,212 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:24,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:24,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:24,214 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,215 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:24,215 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:24,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:24,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:24,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:24,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:24,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:24,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:24,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:24,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:24,289 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420be5fcf436ecd4e4fa4b79a23be6c52b3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420be5fcf436ecd4e4fa4b79a23be6c52b3 2023-04-20 20:54:24,305 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=171, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f4e74f67cc874643ab39c4c75618ad7d 2023-04-20 20:54:24,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024064278, max timestamp=1682024064303Num uniques 859; 2023-04-20 20:54:24,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024064278, max timestamp=1682024064303Num uniques 859; 2023-04-20 20:54:24,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:24,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:24,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024064278, max timestamp=1682024064303Num uniques 859; 2023-04-20 20:54:24,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:24,315 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:24,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:24,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:24,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:24,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:24,322 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:24,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:24,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:24,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/813f51cb5f014cfdb1e8fcee0504ebec 2023-04-20 20:54:24,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,325 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:24,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:24,326 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:24,326 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:24,327 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,327 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:24,327 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:24,327 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:24,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:24,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e185a04[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:24,334 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:24,334 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024124323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024124324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024124325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024124334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:24,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:24,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:24,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:24,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/927bbbcbe87d43d88b44f2d7130329f2 2023-04-20 20:54:24,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f4e74f67cc874643ab39c4c75618ad7d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f4e74f67cc874643ab39c4c75618ad7d 2023-04-20 20:54:24,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:24,347 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:24,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:24,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:24,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:24,347 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:24,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f4e74f67cc874643ab39c4c75618ad7d, entries=200, sequenceid=171, filesize=32.1 K 2023-04-20 20:54:24,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/813f51cb5f014cfdb1e8fcee0504ebec as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/813f51cb5f014cfdb1e8fcee0504ebec 2023-04-20 20:54:24,350 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,350 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,350 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,351 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,356 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:24,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:24,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/813f51cb5f014cfdb1e8fcee0504ebec, entries=150, sequenceid=171, filesize=11.8 K 2023-04-20 20:54:24,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/927bbbcbe87d43d88b44f2d7130329f2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/927bbbcbe87d43d88b44f2d7130329f2 2023-04-20 20:54:24,358 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:24,358 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:24,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/927bbbcbe87d43d88b44f2d7130329f2, entries=150, sequenceid=171, filesize=11.8 K 2023-04-20 20:54:24,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3479312d5cfad95b1d49f86d4ca52095 in 104ms, sequenceid=171, compaction requested=true 2023-04-20 20:54:24,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:24,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:24,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:24,363 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:54:24,363 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:54:24,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:24,364 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:54:24,364 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/B is initiating minor compaction (all files) 2023-04-20 20:54:24,364 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/B in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,365 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/1d460e0c12a84558b152f4235c92a7ae, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e08845360406493e8060a0a41db7948a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/4c8b41f812b94e819336d15c395a2c99, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aefeef38560f44beb30642e509f33acf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/813f51cb5f014cfdb1e8fcee0504ebec] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=58.5 K 2023-04-20 20:54:24,365 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136158 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:54:24,365 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/A is initiating minor compaction (all files) 2023-04-20 20:54:24,365 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/A in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,365 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/312f95734f3743449ec77d878b454124, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/48f5ff774dc44295bd0ed05342a8be87, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/7732968ce84a43b191134e55ea41cd4e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9859022c51ce4140908e6fb5c59726d5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f4e74f67cc874643ab39c4c75618ad7d] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=133.0 K 2023-04-20 20:54:24,365 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1d460e0c12a84558b152f4235c92a7ae, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682024059808 2023-04-20 20:54:24,365 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 312f95734f3743449ec77d878b454124, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682024059808 2023-04-20 20:54:24,366 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e08845360406493e8060a0a41db7948a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682024059936 2023-04-20 20:54:24,366 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 48f5ff774dc44295bd0ed05342a8be87, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682024059936 2023-04-20 20:54:24,366 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4c8b41f812b94e819336d15c395a2c99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1682024061065 2023-04-20 20:54:24,366 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 7732968ce84a43b191134e55ea41cd4e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1682024061065 2023-04-20 20:54:24,367 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting aefeef38560f44beb30642e509f33acf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682024062119 2023-04-20 20:54:24,367 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 813f51cb5f014cfdb1e8fcee0504ebec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682024063193 2023-04-20 20:54:24,368 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9859022c51ce4140908e6fb5c59726d5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682024062119 2023-04-20 20:54:24,368 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f4e74f67cc874643ab39c4c75618ad7d, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682024063193 2023-04-20 20:54:24,378 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#B#compaction#137 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:24,383 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#A#compaction#138 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:24,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:24,439 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=140.89 KB heapSize=400.45 KB 2023-04-20 20:54:24,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024064322, max timestamp=1682024064438Num uniques 1068; 2023-04-20 20:54:24,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024064322, max timestamp=1682024064438Num uniques 1068; 2023-04-20 20:54:24,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-20 20:54:24,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-20 20:54:24,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:24,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:24,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:24,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:54:24,441 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:24,441 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:24,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024124455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,458 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:24,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024124456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024124456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:24,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:24,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:24,461 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:24,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:24,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:24,462 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,462 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024124458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,464 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:24,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:24,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:24,465 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,465 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:24,465 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:24,465 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:24,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:24,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64d169c6[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:24,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:24,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,471 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:24,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:24,475 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:24,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:24,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:24,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:24,475 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:24,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:24,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:24,483 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,483 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,483 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,483 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,483 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:24,483 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:24,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024124559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024124560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024124560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024124564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,583 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:24,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:24,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:24,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:24,587 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:24,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:24,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:24,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:24,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:24,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:24,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:24,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:24,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:24,592 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:24,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68c81c1a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:24,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:24,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,594 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:24,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:24,597 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:24,600 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:24,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:24,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:24,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:24,600 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:24,601 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,601 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,601 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:24,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,605 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:24,606 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:24,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:24,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,706 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:24,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:24,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:24,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:24,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:24,711 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:24,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:24,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:24,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:24,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:24,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:24,715 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,715 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:24,715 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:24,715 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:24,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,716 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:24,716 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:24,718 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,718 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:24,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f7ede4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:24,719 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:24,719 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,720 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:24,721 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,721 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:24,724 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:24,724 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,724 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,724 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:24,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:24,727 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:24,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:24,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:24,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:24,727 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:24,728 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,728 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,728 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:24,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:24,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,733 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:24,733 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:24,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,733 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:24,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024124764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024124765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024124765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:24,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024124767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,790 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/aa50e3d2abf7463292726aacc29d8977 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aa50e3d2abf7463292726aacc29d8977 2023-04-20 20:54:24,798 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/B of 3479312d5cfad95b1d49f86d4ca52095 into aa50e3d2abf7463292726aacc29d8977(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:24,798 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:24,798 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/B, priority=11, startTime=1682024064363; duration=0sec 2023-04-20 20:54:24,798 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:24,798 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:54:24,801 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:54:24,801 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/C is initiating minor compaction (all files) 2023-04-20 20:54:24,801 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/C in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,801 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/4e0ea1b371b54d2d9bb8288ae3955bde, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0347f183a1a64de5bfeedcdf4a7af0ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/45685b5104774c4fb301a817b59d396a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8e2c3d2e3f7749a18406f418fea92e45, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/927bbbcbe87d43d88b44f2d7130329f2] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=58.5 K 2023-04-20 20:54:24,802 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4e0ea1b371b54d2d9bb8288ae3955bde, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682024059808 2023-04-20 20:54:24,802 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0347f183a1a64de5bfeedcdf4a7af0ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682024059936 2023-04-20 20:54:24,803 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 45685b5104774c4fb301a817b59d396a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1682024061065 2023-04-20 20:54:24,803 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8e2c3d2e3f7749a18406f418fea92e45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682024062119 2023-04-20 20:54:24,804 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 927bbbcbe87d43d88b44f2d7130329f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682024063193 2023-04-20 20:54:24,808 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/98206da4305f4dcab156616e53718e5c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/98206da4305f4dcab156616e53718e5c 2023-04-20 20:54:24,817 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/A of 3479312d5cfad95b1d49f86d4ca52095 into 98206da4305f4dcab156616e53718e5c(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:24,817 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:24,817 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/A, priority=11, startTime=1682024064362; duration=0sec 2023-04-20 20:54:24,818 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:24,820 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#C#compaction#140 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:24,833 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:24,835 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/7764c971434b43778f4f443cd7b4d3dc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7764c971434b43778f4f443cd7b4d3dc 2023-04-20 20:54:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:24,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:24,838 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:24,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:24,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:24,839 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,839 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,840 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/C of 3479312d5cfad95b1d49f86d4ca52095 into 7764c971434b43778f4f443cd7b4d3dc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:24,840 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:24,840 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/C, priority=11, startTime=1682024064363; duration=0sec 2023-04-20 20:54:24,840 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:24,841 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,842 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,842 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:24,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:24,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:24,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:24,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:24,844 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:24,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:24,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:24,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:24,845 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:24,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be34d64[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:24,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:24,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,849 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:24,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:24,849 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,849 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:24,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:24,850 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,850 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:24,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:24,855 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:24,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:24,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:24,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:24,856 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:24,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,859 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,860 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420017b21da29524a0dbaa8884dca3b1819 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420017b21da29524a0dbaa8884dca3b1819 2023-04-20 20:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:24,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,864 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:24,865 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:24,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,965 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:24,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:24,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:24,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:24,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:24,968 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:24,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:24,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:24,969 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,969 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:24,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:24,972 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,972 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:24,972 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:24,972 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24938287[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:24,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:24,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:24,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:24,980 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:24,983 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:24,983 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:24,983 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:24,983 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:24,983 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:24,985 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,985 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:24,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,985 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:24,987 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,988 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:24,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:24,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:24,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:24,990 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,990 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:24,990 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:24,990 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:24,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024125069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024125070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024125071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024125072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,091 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:25,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:25,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:25,094 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:25,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:25,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:25,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,096 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:25,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:25,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:25,098 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,098 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:25,098 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:25,098 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:25,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:25,100 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c1372c3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:25,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:25,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,102 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:25,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:25,104 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:25,106 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:25,106 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:25,106 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:25,106 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:25,106 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:25,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:25,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:25,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,120 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:25,120 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:25,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024125206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,209 DEBUG [Thread-816] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:25,220 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:25,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:25,224 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:25,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:25,227 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,227 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:25,227 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:25,227 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:25,229 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a32668[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:25,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:25,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,233 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:25,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:25,237 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:25,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:25,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:25,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:25,237 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:25,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,240 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,242 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:25,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:25,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,243 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:25,243 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:25,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:25,267 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=210, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/11cd1e432754489e88346e75906073d6 2023-04-20 20:54:25,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/fefb875567954c938d9d2cc26bebd899 2023-04-20 20:54:25,343 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:25,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:25,346 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:25,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:25,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:25,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,349 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,349 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,349 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:25,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:25,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:25,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:25,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:25,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:25,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:25,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757ac7f5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:25,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:25,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,355 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:25,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:25,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:25,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:25,360 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:25,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:25,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:25,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:25,360 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:25,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,364 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,366 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,366 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:25,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:25,366 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,366 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,366 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,367 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:25,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,367 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:25,367 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,367 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,367 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,367 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,467 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:25,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:25,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:25,470 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:25,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:25,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:25,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:25,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:25,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:25,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:25,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,476 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc5f467[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:25,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:25,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:25,481 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,481 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:25,487 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:25,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:25,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:25,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:25,487 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:25,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,491 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:25,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:25,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:25,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,495 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:25,495 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:25,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,495 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024125575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024125576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024125577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024125578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,595 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:25,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:25,598 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:25,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:25,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:25,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,600 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:25,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:25,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:25,601 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,601 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:25,602 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:25,602 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:25,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:25,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:25,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad71276[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:25,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,606 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:25,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:25,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:25,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:25,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:25,609 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:25,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:25,611 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:25,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:25,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:25,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:25,611 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:25,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,615 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:25,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:25,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,618 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:25,618 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:25,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/7f9a51a3f98744dcbced1f7f86c6c51e 2023-04-20 20:54:25,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/11cd1e432754489e88346e75906073d6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/11cd1e432754489e88346e75906073d6 2023-04-20 20:54:25,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/11cd1e432754489e88346e75906073d6, entries=200, sequenceid=210, filesize=32.1 K 2023-04-20 20:54:25,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/fefb875567954c938d9d2cc26bebd899 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fefb875567954c938d9d2cc26bebd899 2023-04-20 20:54:25,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fefb875567954c938d9d2cc26bebd899, entries=150, sequenceid=210, filesize=11.8 K 2023-04-20 20:54:25,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/7f9a51a3f98744dcbced1f7f86c6c51e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7f9a51a3f98744dcbced1f7f86c6c51e 2023-04-20 20:54:25,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7f9a51a3f98744dcbced1f7f86c6c51e, entries=150, sequenceid=210, filesize=11.8 K 2023-04-20 20:54:25,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~78.27 KB/80150, heapSize ~213.91 KB/219040, currentSize=100.63 KB/103050 for 3479312d5cfad95b1d49f86d4ca52095 in 1276ms, sequenceid=210, compaction requested=false 2023-04-20 20:54:25,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:25,718 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:25,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:25,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:25,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:25,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:25,721 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:25,721 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,721 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:25,721 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:25,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,723 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:25,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:25,728 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:25,728 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:25,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:25,728 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,728 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:25,728 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 20:54:25,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:25,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:25,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:25,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:25,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:25,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:25,752 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304205b7188509a2e491eb4fef03008d424af to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205b7188509a2e491eb4fef03008d424af 2023-04-20 20:54:25,758 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f7d40973f76e4b979ebe48be9a92f42b 2023-04-20 20:54:25,768 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/beb7c14dcd6e41148200cd7e4eba403d 2023-04-20 20:54:25,786 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/5f9467b5842545f4a62691f02a4abc0f 2023-04-20 20:54:25,793 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f7d40973f76e4b979ebe48be9a92f42b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f7d40973f76e4b979ebe48be9a92f42b 2023-04-20 20:54:25,801 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f7d40973f76e4b979ebe48be9a92f42b, entries=150, sequenceid=231, filesize=25.3 K 2023-04-20 20:54:25,802 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/beb7c14dcd6e41148200cd7e4eba403d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/beb7c14dcd6e41148200cd7e4eba403d 2023-04-20 20:54:25,806 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/beb7c14dcd6e41148200cd7e4eba403d, entries=150, sequenceid=231, filesize=11.8 K 2023-04-20 20:54:25,809 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/5f9467b5842545f4a62691f02a4abc0f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/5f9467b5842545f4a62691f02a4abc0f 2023-04-20 20:54:25,817 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/5f9467b5842545f4a62691f02a4abc0f, entries=150, sequenceid=231, filesize=11.8 K 2023-04-20 20:54:25,819 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 3479312d5cfad95b1d49f86d4ca52095 in 91ms, sequenceid=231, compaction requested=true 2023-04-20 20:54:25,819 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:25,819 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:25,820 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:25,820 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,820 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:25,820 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:25,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,822 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,823 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:25,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@677aff22[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:25,827 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:25,827 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,828 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:25,828 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:25,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,830 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:25,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:25,833 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:25,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:25,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:25,833 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:25,833 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:25,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,836 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,841 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:25,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:25,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,842 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:25,842 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:25,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:25,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,942 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:25,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:25,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:25,947 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:25,947 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,947 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:25,947 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:25,948 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,948 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,949 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,949 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,949 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:25,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:25,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:25,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:25,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:25,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:25,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,954 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449cc957[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:25,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:25,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,958 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:25,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:25,958 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,958 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:25,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:25,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:25,963 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:25,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:25,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:25,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:25,964 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:25,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:25,973 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:25,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:25,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:25,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:25,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:25,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:25,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:25,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:25,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:25,973 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:25,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,973 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:25,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:25,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:25,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,074 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:26,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:26,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:26,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:26,078 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:26,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:26,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:26,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:26,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:26,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:26,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:26,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:26,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:26,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26986b6f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:26,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,088 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:26,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:26,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:26,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:26,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:26,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:26,093 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:26,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:26,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:26,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:26,093 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:26,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:26,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,099 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,100 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:26,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:26,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,100 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:26,200 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:26,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:26,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:26,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:26,205 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:26,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:26,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:26,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,207 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,208 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,208 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:26,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:26,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,209 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:26,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:26,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:26,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:26,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44be8968[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:26,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:26,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,216 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:26,216 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,216 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:26,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:26,218 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:26,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:26,221 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:26,221 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:26,221 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:26,221 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:26,221 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:26,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,228 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:26,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:26,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,231 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:26,231 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:26,331 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:26,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:26,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:26,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:26,344 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:26,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:26,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:26,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:26,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:26,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:26,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:26,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:26,350 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:26,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740ded32[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:26,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:26,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,357 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:26,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:26,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:26,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:26,362 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:26,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:26,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:26,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:26,362 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:26,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,364 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,368 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:26,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:26,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,369 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:26,369 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:26,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:26,469 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:26,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:26,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:26,474 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:26,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:26,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:26,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:26,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:26,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:26,479 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,479 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:26,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:26,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:26,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:26,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56794495[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:26,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:26,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:26,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:26,489 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:26,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:26,492 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:26,493 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:26,493 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:26,493 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:26,493 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:26,496 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,496 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:26,496 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:26,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,501 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,501 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:26,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:26,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:26,502 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:26,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:26,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:26,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:26,503 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:26,503 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:26,603 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:26,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:26,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:26,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:26,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:26,608 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:26,608 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,608 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:26,608 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:26,608 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,610 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:26,610 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:26,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:26,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:26,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:26,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:26,613 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:26,613 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:26,613 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-20 20:54:26,614 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:26,614 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:26,614 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:26,614 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:26,614 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:26,614 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:26,618 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:26,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:26,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682024066617, max timestamp=1682024066645Num uniques 859; 2023-04-20 20:54:26,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682024066617, max timestamp=1682024066645Num uniques 859; 2023-04-20 20:54:26,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682024066617, max timestamp=1682024066645Num uniques 859; 2023-04-20 20:54:26,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-20 20:54:26,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-20 20:54:26,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-20 20:54:26,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024126653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024126654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024126655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024126657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024126757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024126758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024126762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024126766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024126959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024126960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024126964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:26,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:26,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024126969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,043 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420bada722ca70d47ed9b2cc4930778f0da to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bada722ca70d47ed9b2cc4930778f0da 2023-04-20 20:54:27,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024127263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024127264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024127268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024127271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,449 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/499387492fd94834bbee159de9164fa8 2023-04-20 20:54:27,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45358 deadline: 1682024127768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45344 deadline: 1682024127768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024127775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:27,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024127776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:27,862 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/315bfd5d894743ddac76381d3d77ccb5 2023-04-20 20:54:28,273 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/6410650d91c0421bb1aa2f9414e35388 2023-04-20 20:54:28,278 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/499387492fd94834bbee159de9164fa8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/499387492fd94834bbee159de9164fa8 2023-04-20 20:54:28,283 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/499387492fd94834bbee159de9164fa8, entries=250, sequenceid=245, filesize=38.9 K 2023-04-20 20:54:28,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/315bfd5d894743ddac76381d3d77ccb5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/315bfd5d894743ddac76381d3d77ccb5 2023-04-20 20:54:28,289 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/315bfd5d894743ddac76381d3d77ccb5, entries=150, sequenceid=245, filesize=11.8 K 2023-04-20 20:54:28,290 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/6410650d91c0421bb1aa2f9414e35388 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6410650d91c0421bb1aa2f9414e35388 2023-04-20 20:54:28,294 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6410650d91c0421bb1aa2f9414e35388, entries=150, sequenceid=245, filesize=11.8 K 2023-04-20 20:54:28,295 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=161.02 KB/164880 for 3479312d5cfad95b1d49f86d4ca52095 in 1682ms, sequenceid=245, compaction requested=true 2023-04-20 20:54:28,295 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:28,295 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:28,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:28,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:28,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335f8e26[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:28,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:28,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:28,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:28,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:28,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:28,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:28,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:28,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:28,306 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:28,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:28,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:28,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:28,306 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:28,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:28,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:28,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:28,315 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:28,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:28,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:28,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,316 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:28,316 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:28,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:28,416 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:28,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:28,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:28,419 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:28,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:28,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:28,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:28,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,422 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,423 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,423 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:28,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:28,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:28,424 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:28,424 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:28,425 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=161.02 KB heapSize=399.05 KB 2023-04-20 20:54:28,425 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:28,425 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:28,425 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:28,425 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:28,425 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:28,425 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:28,448 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420dce17f81ded64f00af472fcc62526069 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420dce17f81ded64f00af472fcc62526069 2023-04-20 20:54:28,455 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f59c029190114a8ca851928c61d3d786 2023-04-20 20:54:28,477 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/698fc42038de4c1f982b88062d71de61 2023-04-20 20:54:28,501 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/c9bc35ee17d24c81b219ccc55ef24d19 2023-04-20 20:54:28,506 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/f59c029190114a8ca851928c61d3d786 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f59c029190114a8ca851928c61d3d786 2023-04-20 20:54:28,513 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f59c029190114a8ca851928c61d3d786, entries=150, sequenceid=272, filesize=25.4 K 2023-04-20 20:54:28,514 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/698fc42038de4c1f982b88062d71de61 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/698fc42038de4c1f982b88062d71de61 2023-04-20 20:54:28,518 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/698fc42038de4c1f982b88062d71de61, entries=150, sequenceid=272, filesize=11.9 K 2023-04-20 20:54:28,519 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/c9bc35ee17d24c81b219ccc55ef24d19 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/c9bc35ee17d24c81b219ccc55ef24d19 2023-04-20 20:54:28,525 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/c9bc35ee17d24c81b219ccc55ef24d19, entries=150, sequenceid=272, filesize=11.9 K 2023-04-20 20:54:28,526 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~399 KB/408576, currentSize=0 B/0 for 3479312d5cfad95b1d49f86d4ca52095 in 101ms, sequenceid=272, compaction requested=true 2023-04-20 20:54:28,526 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:28,526 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,529 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,529 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31ab4ec[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:28,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:28,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,532 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:28,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:28,533 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,533 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:28,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:28,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:28,538 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:28,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:28,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:28,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:28,538 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:28,540 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,540 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,540 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:28,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,548 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:28,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:28,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:28,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:28,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:28,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:28,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,549 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:28,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:28,549 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:28,649 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:28,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:28,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:28,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:28,653 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:28,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:28,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:28,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:28,654 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,654 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,655 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:28,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:28,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:28,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:28,657 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:28,658 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:28,658 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:28,658 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:28,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:28,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:28,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:28,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:28,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,661 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,661 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,661 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:28,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1865bdc0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:28,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:28,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,665 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:28,666 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,666 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:28,670 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:28,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:28,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:28,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:28,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:28,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:28,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:28,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:28,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:28,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:28,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:28,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:28,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:28,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:28,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:28,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:28,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,681 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:28,681 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:28,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:28,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:28,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,781 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:28,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:28,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:28,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:28,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:28,789 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:28,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:28,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:28,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:28,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,792 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:28,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:28,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:28,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:28,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:28,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:28,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:28,795 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-20 20:54:28,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:28,796 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:28,796 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:28,796 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:28,796 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:28,796 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:28,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:28,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:28,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024068839, max timestamp=1682024068862Num uniques 859; 2023-04-20 20:54:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-20 20:54:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682024068839, max timestamp=1682024068862Num uniques 859; 2023-04-20 20:54:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682024068839, max timestamp=1682024068862Num uniques 859; 2023-04-20 20:54:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:28,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:28,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1682024068909, max timestamp=1682024068921Num uniques 1009; 2023-04-20 20:54:28,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=21428, suffix heap size=132904,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=468 2023-04-20 20:54:28,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1682024068909, max timestamp=1682024068921Num uniques 1068; 2023-04-20 20:54:28,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=21428, suffix heap size=139040,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=468 2023-04-20 20:54:28,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1682024068909, max timestamp=1682024068921Num uniques 1068; 2023-04-20 20:54:28,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=21428, suffix heap size=139040,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=468 2023-04-20 20:54:28,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024128938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024128939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024128940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:28,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:28,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024128943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024129044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024129044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024129045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024129047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,243 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420bcb1c2aa94d9475ba880b2db00a87842 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bcb1c2aa94d9475ba880b2db00a87842 2023-04-20 20:54:29,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45392 deadline: 1682024129246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,249 DEBUG [Thread-816] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:29,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024129250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024129250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024129250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024129253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,260 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=278, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/89a3500afdb54e9ebb7c77ce83cd518d 2023-04-20 20:54:29,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024129553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45368 deadline: 1682024129553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024129556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:29,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45330 deadline: 1682024129560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:29,673 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/783d78e718724f45b1dacb5990df5d9d 2023-04-20 20:54:30,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:30,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024130055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024130059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024130059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:30,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45330 deadline: 1682024130065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,089 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/8a4dc42800c049ee80a7998b1a2448ef 2023-04-20 20:54:30,094 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/89a3500afdb54e9ebb7c77ce83cd518d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/89a3500afdb54e9ebb7c77ce83cd518d 2023-04-20 20:54:30,098 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/89a3500afdb54e9ebb7c77ce83cd518d, entries=100, sequenceid=278, filesize=18.6 K 2023-04-20 20:54:30,099 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/783d78e718724f45b1dacb5990df5d9d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/783d78e718724f45b1dacb5990df5d9d 2023-04-20 20:54:30,102 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/783d78e718724f45b1dacb5990df5d9d, entries=100, sequenceid=278, filesize=9.5 K 2023-04-20 20:54:30,103 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/8a4dc42800c049ee80a7998b1a2448ef as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8a4dc42800c049ee80a7998b1a2448ef 2023-04-20 20:54:30,108 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8a4dc42800c049ee80a7998b1a2448ef, entries=100, sequenceid=278, filesize=9.5 K 2023-04-20 20:54:30,108 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=174.43 KB/178620 for 3479312d5cfad95b1d49f86d4ca52095 in 1313ms, sequenceid=278, compaction requested=true 2023-04-20 20:54:30,108 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:30,108 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:30,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:30,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:30,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:30,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:30,116 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4241fb5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:30,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:30,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:30,119 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,119 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:30,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:30,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:30,124 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:30,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:30,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:30,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:30,124 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:30,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:30,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:30,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:30,128 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:30,132 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,135 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:30,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:30,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:30,135 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:30,135 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:30,135 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,135 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:30,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:30,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:30,135 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:30,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:30,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:30,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:30,235 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:30,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:30,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:30,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:30,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:30,239 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:30,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:30,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:30,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:30,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:30,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,241 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:30,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:30,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:30,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:30,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:30,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:30,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:30,242 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:30,243 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=174.43 KB heapSize=475.55 KB 2023-04-20 20:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:30,658 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304202802a7cb6fb24dd7a35630ddf1dbf043 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202802a7cb6fb24dd7a35630ddf1dbf043 2023-04-20 20:54:31,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:31,062 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:31,065 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=332, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/2cea0b8e95d040ceb4ba71f50d37c8ee 2023-04-20 20:54:31,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:31,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024131063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:31,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024131064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:31,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024131072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:31,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45330 deadline: 1682024131078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:31,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024131167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:31,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024131372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,479 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/7db67099cee443daba5abb5017a4e409 2023-04-20 20:54:31,490 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/7905f96943474d95b22f8f393a48d987 2023-04-20 20:54:31,496 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/2cea0b8e95d040ceb4ba71f50d37c8ee as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/2cea0b8e95d040ceb4ba71f50d37c8ee 2023-04-20 20:54:31,500 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/2cea0b8e95d040ceb4ba71f50d37c8ee, entries=150, sequenceid=332, filesize=25.4 K 2023-04-20 20:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/7db67099cee443daba5abb5017a4e409 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/7db67099cee443daba5abb5017a4e409 2023-04-20 20:54:31,506 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/7db67099cee443daba5abb5017a4e409, entries=150, sequenceid=332, filesize=11.9 K 2023-04-20 20:54:31,507 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/7905f96943474d95b22f8f393a48d987 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7905f96943474d95b22f8f393a48d987 2023-04-20 20:54:31,511 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7905f96943474d95b22f8f393a48d987, entries=150, sequenceid=332, filesize=11.9 K 2023-04-20 20:54:31,512 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~475.50 KB/486912, currentSize=13.42 KB/13740 for 3479312d5cfad95b1d49f86d4ca52095 in 1269ms, sequenceid=332, compaction requested=true 2023-04-20 20:54:31,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:31,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:31,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:31,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:31,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:31,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:31,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,517 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:31,518 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,518 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:31,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:31,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bc59fa9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:31,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:31,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,521 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:31,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:31,523 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:31,538 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:31,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:31,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:31,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:31,538 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:31,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:31,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:31,540 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:31,541 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:31,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:31,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:31,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:31,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:31,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:31,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:31,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:31,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:31,544 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:31,544 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,544 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:31,544 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,544 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:31,544 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:31,644 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:31,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:31,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:31,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:31,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:31,663 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:31,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:31,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:31,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:31,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:31,666 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,666 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:31,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:31,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:31,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:31,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:31,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:31,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:31,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:31,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:31,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:31,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:31,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:31,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:31,670 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:31,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:31,708 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304200369ddc96a9a4fb0ae42c59d9313dd53 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200369ddc96a9a4fb0ae42c59d9313dd53 2023-04-20 20:54:31,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:31,733 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:31,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024071767, max timestamp=1682024071785Num uniques 859; 2023-04-20 20:54:31,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:31,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024071767, max timestamp=1682024071785Num uniques 859; 2023-04-20 20:54:31,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:31,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024071767, max timestamp=1682024071785Num uniques 859; 2023-04-20 20:54:31,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:31,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024071838, max timestamp=1682024071856Num uniques 1018; 2023-04-20 20:54:31,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:54:31,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024071838, max timestamp=1682024071856Num uniques 1018; 2023-04-20 20:54:31,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:54:31,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024071838, max timestamp=1682024071856Num uniques 1018; 2023-04-20 20:54:31,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:54:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682024071893, max timestamp=1682024071915Num uniques 1077; 2023-04-20 20:54:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682024071893, max timestamp=1682024071915Num uniques 1077; 2023-04-20 20:54:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-20 20:54:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682024071893, max timestamp=1682024071915Num uniques 1077; 2023-04-20 20:54:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-20 20:54:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-20 20:54:31,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682024071965, max timestamp=1682024071992Num uniques 1136; 2023-04-20 20:54:31,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-20 20:54:31,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682024071965, max timestamp=1682024071992Num uniques 1136; 2023-04-20 20:54:31,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682024071965, max timestamp=1682024071992Num uniques 1136; 2023-04-20 20:54:31,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-20 20:54:31,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-20 20:54:32,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024072058, max timestamp=1682024072081Num uniques 1045; 2023-04-20 20:54:32,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-20 20:54:32,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024072058, max timestamp=1682024072081Num uniques 1045; 2023-04-20 20:54:32,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-20 20:54:32,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024072058, max timestamp=1682024072081Num uniques 1045; 2023-04-20 20:54:32,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-20 20:54:32,126 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/0f945371ea03493d881987d029749e7a 2023-04-20 20:54:32,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024072150, max timestamp=1682024072165Num uniques 1054; 2023-04-20 20:54:32,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:54:32,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024072150, max timestamp=1682024072165Num uniques 1054; 2023-04-20 20:54:32,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:54:32,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024072150, max timestamp=1682024072165Num uniques 1054; 2023-04-20 20:54:32,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:54:32,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682024072216, max timestamp=1682024072235Num uniques 1013; 2023-04-20 20:54:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-20 20:54:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682024072216, max timestamp=1682024072235Num uniques 1013; 2023-04-20 20:54:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-20 20:54:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682024072216, max timestamp=1682024072235Num uniques 1013; 2023-04-20 20:54:32,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-20 20:54:32,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024072285, max timestamp=1682024072313Num uniques 1022; 2023-04-20 20:54:32,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-20 20:54:32,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024072285, max timestamp=1682024072313Num uniques 1022; 2023-04-20 20:54:32,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-20 20:54:32,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024072285, max timestamp=1682024072313Num uniques 1022; 2023-04-20 20:54:32,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-20 20:54:32,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024072372, max timestamp=1682024072393Num uniques 1131; 2023-04-20 20:54:32,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-20 20:54:32,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024072372, max timestamp=1682024072393Num uniques 1131; 2023-04-20 20:54:32,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-20 20:54:32,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024072372, max timestamp=1682024072393Num uniques 1131; 2023-04-20 20:54:32,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-20 20:54:32,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1682024072423, max timestamp=1682024072447Num uniques 1090; 2023-04-20 20:54:32,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-20 20:54:32,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1682024072423, max timestamp=1682024072447Num uniques 1090; 2023-04-20 20:54:32,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-20 20:54:32,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1682024072423, max timestamp=1682024072447Num uniques 1090; 2023-04-20 20:54:32,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-20 20:54:32,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682024072500, max timestamp=1682024072512Num uniques 1199; 2023-04-20 20:54:32,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-20 20:54:32,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682024072500, max timestamp=1682024072512Num uniques 1199; 2023-04-20 20:54:32,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-20 20:54:32,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682024072500, max timestamp=1682024072512Num uniques 1199; 2023-04-20 20:54:32,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-20 20:54:32,539 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/6bfdb2b9915d46feac6a61554e88483d 2023-04-20 20:54:32,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682024072552, max timestamp=1682024072582Num uniques 1058; 2023-04-20 20:54:32,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682024072552, max timestamp=1682024072582Num uniques 1058; 2023-04-20 20:54:32,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682024072552, max timestamp=1682024072582Num uniques 1058; 2023-04-20 20:54:32,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-20 20:54:32,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-20 20:54:32,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-20 20:54:32,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024072622, max timestamp=1682024072640Num uniques 1017; 2023-04-20 20:54:32,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-20 20:54:32,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024072622, max timestamp=1682024072640Num uniques 1017; 2023-04-20 20:54:32,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024072622, max timestamp=1682024072640Num uniques 1017; 2023-04-20 20:54:32,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-20 20:54:32,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-20 20:54:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682024072677, max timestamp=1682024072694Num uniques 1076; 2023-04-20 20:54:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682024072677, max timestamp=1682024072694Num uniques 1076; 2023-04-20 20:54:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682024072677, max timestamp=1682024072694Num uniques 1076; 2023-04-20 20:54:32,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-20 20:54:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-20 20:54:32,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-20 20:54:32,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024072740, max timestamp=1682024072754Num uniques 1085; 2023-04-20 20:54:32,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-20 20:54:32,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024072740, max timestamp=1682024072754Num uniques 1085; 2023-04-20 20:54:32,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024072740, max timestamp=1682024072754Num uniques 1085; 2023-04-20 20:54:32,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-20 20:54:32,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-20 20:54:32,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682024072779, max timestamp=1682024072827Num uniques 1094; 2023-04-20 20:54:32,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682024072779, max timestamp=1682024072827Num uniques 1094; 2023-04-20 20:54:32,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-20 20:54:32,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-20 20:54:32,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682024072779, max timestamp=1682024072827Num uniques 1094; 2023-04-20 20:54:32,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-20 20:54:32,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1682024072871, max timestamp=1682024072901Num uniques 1103; 2023-04-20 20:54:32,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=13875, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-04-20 20:54:32,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1682024072871, max timestamp=1682024072901Num uniques 1103; 2023-04-20 20:54:32,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1682024072871, max timestamp=1682024072901Num uniques 1103; 2023-04-20 20:54:32,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=13875, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-04-20 20:54:32,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=13875, suffix heap size=142680,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=303 2023-04-20 20:54:32,950 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/b941c030e4784d269f349f484cfcd3bc 2023-04-20 20:54:32,954 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/0f945371ea03493d881987d029749e7a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/0f945371ea03493d881987d029749e7a 2023-04-20 20:54:32,958 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/0f945371ea03493d881987d029749e7a, entries=100, sequenceid=337, filesize=18.6 K 2023-04-20 20:54:32,959 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/6bfdb2b9915d46feac6a61554e88483d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6bfdb2b9915d46feac6a61554e88483d 2023-04-20 20:54:32,962 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6bfdb2b9915d46feac6a61554e88483d, entries=100, sequenceid=337, filesize=9.5 K 2023-04-20 20:54:32,964 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/b941c030e4784d269f349f484cfcd3bc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b941c030e4784d269f349f484cfcd3bc 2023-04-20 20:54:32,969 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b941c030e4784d269f349f484cfcd3bc, entries=100, sequenceid=337, filesize=9.5 K 2023-04-20 20:54:32,969 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=147.60 KB/151140 for 3479312d5cfad95b1d49f86d4ca52095 in 1299ms, sequenceid=337, compaction requested=true 2023-04-20 20:54:32,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:32,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:32,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d2effe9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:32,973 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:32,973 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:32,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:32,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:32,976 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:32,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=154.31 KB heapSize=431.25 KB 2023-04-20 20:54:32,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:32,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:32,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:32,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:32,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:32,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:32,978 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:32,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:32,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:32,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:32,978 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:32,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:32,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:32,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:32,988 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:32,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:32,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:32,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:32,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:32,996 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:32,995 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:32,996 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:32,996 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:32,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:32,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,000 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420e6dce45d3fa64823aace075348537a4c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e6dce45d3fa64823aace075348537a4c 2023-04-20 20:54:33,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:33,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024133013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:33,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024133076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,085 DEBUG [Thread-814] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:33,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:33,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45330 deadline: 1682024133087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,092 DEBUG [Thread-820] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:33,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:33,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024133089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,097 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:33,097 DEBUG [Thread-818] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:33,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:33,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:33,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:33,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:33,101 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:33,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:33,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:33,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,102 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,102 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:33,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:33,104 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,104 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:33,105 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:33,105 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:33,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:33,106 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2810ca31[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:33,108 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:33,108 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,110 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:33,110 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,110 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:33,110 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:33,112 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:33,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:33,114 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:33,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:33,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:33,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:33,115 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:33,116 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,116 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,116 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,117 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:33,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024133117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:33,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:33,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,121 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:33,121 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:33,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:33,221 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:33,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:33,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:33,225 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:33,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:33,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:33,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,227 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:33,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:33,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:33,229 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:33,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:33,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c6db131[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:33,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:33,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,233 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:33,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:33,235 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:33,237 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:33,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:33,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:33,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:33,238 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:33,243 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,244 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,244 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,244 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,247 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,247 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,247 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,248 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:33,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:33,248 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:33,249 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:33,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:33,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024133319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,349 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:33,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:33,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:33,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:33,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:33,352 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:33,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:33,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:33,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,354 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:33,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:33,356 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,356 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:33,356 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:33,356 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:33,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:33,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b325fc2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:33,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:33,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:33,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:33,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:33,367 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:33,367 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:33,367 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:33,367 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:33,367 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:33,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,369 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,371 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:33,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:33,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,377 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,377 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,378 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:33,405 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=649, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/6de72be12e994a19a38107d3cf9f6494 2023-04-20 20:54:33,478 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:33,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:33,481 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:33,481 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,481 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:33,481 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:33,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,483 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,483 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,483 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,483 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:33,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:33,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:33,485 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,485 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:33,485 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:33,485 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:33,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:33,487 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,487 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,487 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:33,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18934fa[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:33,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:33,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,490 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,490 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,490 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:33,490 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:33,490 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:33,490 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:33,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:33,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:33,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:33,494 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:33,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:33,495 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:33,495 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:33,495 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:33,496 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,496 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,496 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,497 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:33,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:33,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:33,500 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:33,501 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:33,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,601 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:33,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:33,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:33,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:33,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:33,604 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:33,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:33,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:33,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,606 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:33,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:33,608 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:33,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:33,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe637b8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:33,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:33,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:33,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:33,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:33,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:33,615 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:33,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:33,617 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:33,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:33,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:33,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:33,617 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:33,619 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,619 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,619 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,623 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:33,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,624 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:33,624 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:33,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:33,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:33,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024133624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,724 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:33,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:33,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:33,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:33,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:33,727 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:33,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:33,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:33,728 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,728 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,729 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,729 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:33,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:33,731 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,731 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:33,731 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:33,731 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:33,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,733 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,733 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e63ef4c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:33,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:33,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:33,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:33,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:33,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:33,737 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,737 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:33,737 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,737 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:33,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:33,740 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:33,740 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:33,740 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:33,740 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:33,740 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:33,742 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,742 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:33,742 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:33,743 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:33,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:33,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:33,748 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:33,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,748 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:33,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:33,816 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/61433609442a4775b9a02326c7531c35 2023-04-20 20:54:33,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=649 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/55291c51494e4d388d05435911fb5fed 2023-04-20 20:54:33,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/6de72be12e994a19a38107d3cf9f6494 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/6de72be12e994a19a38107d3cf9f6494 2023-04-20 20:54:33,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/6de72be12e994a19a38107d3cf9f6494, entries=250, sequenceid=649, filesize=39.2 K 2023-04-20 20:54:33,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/61433609442a4775b9a02326c7531c35 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/61433609442a4775b9a02326c7531c35 2023-04-20 20:54:33,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/61433609442a4775b9a02326c7531c35, entries=150, sequenceid=649, filesize=11.9 K 2023-04-20 20:54:33,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/55291c51494e4d388d05435911fb5fed as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/55291c51494e4d388d05435911fb5fed 2023-04-20 20:54:33,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/55291c51494e4d388d05435911fb5fed, entries=150, sequenceid=649, filesize=11.9 K 2023-04-20 20:54:33,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.20 KB/441552, currentSize=33.54 KB/34350 for 3479312d5cfad95b1d49f86d4ca52095 in 868ms, sequenceid=649, compaction requested=true 2023-04-20 20:54:33,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:33,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:33,845 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-20 20:54:33,845 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-20 20:54:33,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:33,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:33,848 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 255084 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-20 20:54:33,848 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/A is initiating minor compaction (all files) 2023-04-20 20:54:33,848 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/A in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,848 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/98206da4305f4dcab156616e53718e5c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/11cd1e432754489e88346e75906073d6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f7d40973f76e4b979ebe48be9a92f42b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/499387492fd94834bbee159de9164fa8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f59c029190114a8ca851928c61d3d786, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/89a3500afdb54e9ebb7c77ce83cd518d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/2cea0b8e95d040ceb4ba71f50d37c8ee, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/0f945371ea03493d881987d029749e7a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/6de72be12e994a19a38107d3cf9f6494] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=249.1 K 2023-04-20 20:54:33,849 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:33,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 104494 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-20 20:54:33,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/B is initiating minor compaction (all files) 2023-04-20 20:54:33,849 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/B in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,849 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aa50e3d2abf7463292726aacc29d8977, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fefb875567954c938d9d2cc26bebd899, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/beb7c14dcd6e41148200cd7e4eba403d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/315bfd5d894743ddac76381d3d77ccb5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/698fc42038de4c1f982b88062d71de61, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/783d78e718724f45b1dacb5990df5d9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/7db67099cee443daba5abb5017a4e409, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6bfdb2b9915d46feac6a61554e88483d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/61433609442a4775b9a02326c7531c35] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=102.0 K 2023-04-20 20:54:33,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:33,849 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 98206da4305f4dcab156616e53718e5c, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682024063193 2023-04-20 20:54:33,850 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting aa50e3d2abf7463292726aacc29d8977, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682024063193 2023-04-20 20:54:33,850 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 11cd1e432754489e88346e75906073d6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682024064322 2023-04-20 20:54:33,850 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting fefb875567954c938d9d2cc26bebd899, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682024064322 2023-04-20 20:54:33,850 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f7d40973f76e4b979ebe48be9a92f42b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682024064444 2023-04-20 20:54:33,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting beb7c14dcd6e41148200cd7e4eba403d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682024064444 2023-04-20 20:54:33,851 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 499387492fd94834bbee159de9164fa8, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682024066604 2023-04-20 20:54:33,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 315bfd5d894743ddac76381d3d77ccb5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682024066604 2023-04-20 20:54:33,851 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f59c029190114a8ca851928c61d3d786, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682024066646 2023-04-20 20:54:33,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 698fc42038de4c1f982b88062d71de61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682024066646 2023-04-20 20:54:33,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:33,852 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 89a3500afdb54e9ebb7c77ce83cd518d, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682024068781 2023-04-20 20:54:33,852 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 783d78e718724f45b1dacb5990df5d9d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682024068781 2023-04-20 20:54:33,852 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2cea0b8e95d040ceb4ba71f50d37c8ee, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682024068938 2023-04-20 20:54:33,852 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7db67099cee443daba5abb5017a4e409, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682024068938 2023-04-20 20:54:33,853 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0f945371ea03493d881987d029749e7a, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682024071061 2023-04-20 20:54:33,853 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 6bfdb2b9915d46feac6a61554e88483d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682024071061 2023-04-20 20:54:33,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:33,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:33,853 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:33,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:33,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:33,853 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6de72be12e994a19a38107d3cf9f6494, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682024072946 2023-04-20 20:54:33,853 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 61433609442a4775b9a02326c7531c35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682024072946 2023-04-20 20:54:33,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,854 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,856 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:33,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:33,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:33,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:33,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:33,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:33,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:33,857 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:33,857 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:33,857 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:33,858 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:33,858 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:33,858 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:33,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:33,892 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#B#compaction#166 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:33,903 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#A#compaction#165 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:33,918 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/e6a2a12136964ad4af422e5eb900ef3e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e6a2a12136964ad4af422e5eb900ef3e 2023-04-20 20:54:33,922 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/B of 3479312d5cfad95b1d49f86d4ca52095 into e6a2a12136964ad4af422e5eb900ef3e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:33,922 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:33,922 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/B, priority=7, startTime=1682024073845; duration=0sec 2023-04-20 20:54:33,923 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:33,923 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-20 20:54:33,925 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 104494 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-20 20:54:33,925 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/C is initiating minor compaction (all files) 2023-04-20 20:54:33,925 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/C in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:33,926 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7764c971434b43778f4f443cd7b4d3dc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7f9a51a3f98744dcbced1f7f86c6c51e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/5f9467b5842545f4a62691f02a4abc0f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6410650d91c0421bb1aa2f9414e35388, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/c9bc35ee17d24c81b219ccc55ef24d19, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8a4dc42800c049ee80a7998b1a2448ef, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7905f96943474d95b22f8f393a48d987, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b941c030e4784d269f349f484cfcd3bc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/55291c51494e4d388d05435911fb5fed] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=102.0 K 2023-04-20 20:54:33,926 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7764c971434b43778f4f443cd7b4d3dc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682024063193 2023-04-20 20:54:33,926 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7f9a51a3f98744dcbced1f7f86c6c51e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682024064322 2023-04-20 20:54:33,927 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5f9467b5842545f4a62691f02a4abc0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682024064444 2023-04-20 20:54:33,927 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 6410650d91c0421bb1aa2f9414e35388, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682024066604 2023-04-20 20:54:33,927 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c9bc35ee17d24c81b219ccc55ef24d19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682024066646 2023-04-20 20:54:33,927 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8a4dc42800c049ee80a7998b1a2448ef, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682024068781 2023-04-20 20:54:33,928 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7905f96943474d95b22f8f393a48d987, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682024068938 2023-04-20 20:54:33,928 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b941c030e4784d269f349f484cfcd3bc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682024071061 2023-04-20 20:54:33,930 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 55291c51494e4d388d05435911fb5fed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682024072946 2023-04-20 20:54:33,949 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#C#compaction#167 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:34,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:34,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:34,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024074174, max timestamp=1682024074190Num uniques 859; 2023-04-20 20:54:34,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024074174, max timestamp=1682024074190Num uniques 859; 2023-04-20 20:54:34,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024074174, max timestamp=1682024074190Num uniques 859; 2023-04-20 20:54:34,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:34,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:34,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:54:34,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024074203, max timestamp=1682024074255Num uniques 1068; 2023-04-20 20:54:34,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024074203, max timestamp=1682024074255Num uniques 1068; 2023-04-20 20:54:34,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024074203, max timestamp=1682024074255Num uniques 1068; 2023-04-20 20:54:34,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:54:34,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:54:34,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:54:34,286 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420a0261cdef95b49619cd329f24b90818b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a0261cdef95b49619cd329f24b90818b 2023-04-20 20:54:34,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024074288, max timestamp=1682024074331Num uniques 1027; 2023-04-20 20:54:34,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-20 20:54:34,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024074288, max timestamp=1682024074331Num uniques 1027; 2023-04-20 20:54:34,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024074288, max timestamp=1682024074331Num uniques 1027; 2023-04-20 20:54:34,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-20 20:54:34,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-20 20:54:34,359 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/0d30d2e79d7a43d6b39ca4749dda0259 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0d30d2e79d7a43d6b39ca4749dda0259 2023-04-20 20:54:34,363 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/C of 3479312d5cfad95b1d49f86d4ca52095 into 0d30d2e79d7a43d6b39ca4749dda0259(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:34,363 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:34,363 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/C, priority=7, startTime=1682024073846; duration=0sec 2023-04-20 20:54:34,364 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:34,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682024074382, max timestamp=1682024074402Num uniques 1086; 2023-04-20 20:54:34,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682024074382, max timestamp=1682024074402Num uniques 1086; 2023-04-20 20:54:34,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-20 20:54:34,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-20 20:54:34,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024074393, max timestamp=1682024074402Num uniques 1086; 2023-04-20 20:54:34,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=6870, suffix heap size=140912,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=150 2023-04-20 20:54:34,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682024074433, max timestamp=1682024074451Num uniques 1145; 2023-04-20 20:54:34,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682024074433, max timestamp=1682024074451Num uniques 1009; 2023-04-20 20:54:34,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682024074433, max timestamp=1682024074451Num uniques 1145; 2023-04-20 20:54:34,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13510, suffix heap size=132904,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=295 2023-04-20 20:54:34,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-20 20:54:34,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-20 20:54:34,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:34,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 563 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024134499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:34,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:34,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 565 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024134603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:34,692 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=657, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/ee56db696a0d4894b3ed3d41ad926c05 2023-04-20 20:54:34,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:34,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 567 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024134808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,104 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/244374fa7f6e494ebb58c0028900e6fa 2023-04-20 20:54:35,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:35,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 569 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024135113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,126 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/e47f239ab5a247a88ecce8a00d45d998 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/e47f239ab5a247a88ecce8a00d45d998 2023-04-20 20:54:35,136 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/A of 3479312d5cfad95b1d49f86d4ca52095 into e47f239ab5a247a88ecce8a00d45d998(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-20 20:54:35,136 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:35,136 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/A, priority=7, startTime=1682024073845; duration=1sec 2023-04-20 20:54:35,137 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:35,519 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/b601591eee32441ab09275c17d72b503 2023-04-20 20:54:35,524 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/ee56db696a0d4894b3ed3d41ad926c05 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ee56db696a0d4894b3ed3d41ad926c05 2023-04-20 20:54:35,527 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ee56db696a0d4894b3ed3d41ad926c05, entries=150, sequenceid=657, filesize=25.4 K 2023-04-20 20:54:35,528 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/244374fa7f6e494ebb58c0028900e6fa as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/244374fa7f6e494ebb58c0028900e6fa 2023-04-20 20:54:35,532 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/244374fa7f6e494ebb58c0028900e6fa, entries=150, sequenceid=657, filesize=11.9 K 2023-04-20 20:54:35,533 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/b601591eee32441ab09275c17d72b503 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b601591eee32441ab09275c17d72b503 2023-04-20 20:54:35,537 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b601591eee32441ab09275c17d72b503, entries=150, sequenceid=657, filesize=11.9 K 2023-04-20 20:54:35,538 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 3479312d5cfad95b1d49f86d4ca52095 in 1679ms, sequenceid=657, compaction requested=false 2023-04-20 20:54:35,538 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:35,538 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:35,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:35,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:35,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:35,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:35,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:35,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3756650d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:35,541 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:35,541 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,542 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:35,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:35,544 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:35,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:35,546 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:35,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:35,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:35,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:35,547 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:35,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,550 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,552 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:35,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:35,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,554 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:35,554 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:35,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:35,620 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-20 20:54:35,620 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:35,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:35,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:35,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:35,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:35,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:35,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:35,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 576 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024135632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,655 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:35,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:35,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:35,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:35,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:35,658 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:35,658 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:35,658 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:35,658 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:35,659 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,659 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:35,660 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:35,661 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,661 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:35,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:35,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:35,662 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:35,662 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:35,662 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:35,662 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:35,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,663 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:35,663 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:35,665 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:35,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f30b9f9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:35,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:35,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,669 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:35,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:35,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:35,670 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:35,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:35,677 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:35,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:35,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:35,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:35,677 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:35,678 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,681 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,683 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:35,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:35,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:35,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:35,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,686 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:35,686 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:35,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 578 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024135737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,786 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:35,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:35,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:35,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:35,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:35,790 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:35,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:35,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:35,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:35,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,793 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:35,793 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,793 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,793 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:35,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:35,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:35,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:35,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:35,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:35,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:35,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:35,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:35,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:35,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:35,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:35,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:35,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:35,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,797 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,798 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:35,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61750a50[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:35,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:35,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:35,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:35,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:35,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:35,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:35,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:35,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:35,803 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:35,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:35,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:35,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:35,803 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:35,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,807 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,814 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,814 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:35,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:35,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,814 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:35,814 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:35,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:35,914 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:35,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:35,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:35,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:35,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:35,918 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:35,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:35,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:35,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:35,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,919 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:35,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:35,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:35,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:35,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:35,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:35,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:35,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:35,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:35,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:35,926 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:35,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@303d80bd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:35,928 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:35,928 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:35,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:35,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:35,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:35,934 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:35,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:35,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:35,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:35,934 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:35,937 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,937 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:35,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,937 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:35,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:35,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:35,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:35,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:35,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:35,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:35,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:35,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:35,942 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:35,942 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:35,942 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,942 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:35,942 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:35,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:35,942 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:35,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:35,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 580 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024135941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,036 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420f5e81aba03b1484499a727da2fc003d3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420f5e81aba03b1484499a727da2fc003d3 2023-04-20 20:54:36,042 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:36,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:36,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:36,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:36,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:36,047 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:36,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:36,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:36,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:36,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,049 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:36,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:36,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:36,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:36,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:36,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:36,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:36,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:36,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:36,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:36,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:36,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:36,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:36,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:36,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:36,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:36,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:36,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,053 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3447e5ba[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:36,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:36,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,057 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:36,057 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:36,057 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:36,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:36,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:36,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:36,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:36,061 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:36,061 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:36,061 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:36,061 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:36,061 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:36,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:36,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,063 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:36,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:36,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:36,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,068 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,068 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,068 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:36,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:36,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:36,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:36,068 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:36,068 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:36,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:36,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:36,168 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:36,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:36,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:36,171 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:36,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:36,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:36,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:36,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:36,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:36,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:36,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:36,176 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:36,176 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:36,176 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:36,176 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:36,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:36,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:36,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:36,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,179 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,179 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec27dba[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:36,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:36,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,181 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,181 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:36,181 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:36,181 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:36,181 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:36,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:36,183 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:36,185 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:36,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:36,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:36,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:36,186 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:36,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:36,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,189 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:36,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:36,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:36,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,192 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:36,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,192 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:36,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:36,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:36,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:36,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 582 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024136247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,293 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:36,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:36,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:36,296 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:36,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:36,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:36,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:36,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:36,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:36,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:36,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:36,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:36,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:36,299 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:36,299 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. started... 2023-04-20 20:54:36,300 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. as already flushing 2023-04-20 20:54:36,300 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:36,301 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:36,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7acc9b8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:36,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:36,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,307 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:36,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:36,309 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,309 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:36,309 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,309 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:36,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:36,311 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:36,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:36,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:36,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:36,311 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:36,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:36,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:36,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:36,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:36,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:36,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:36,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:36,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:36,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:36,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:36,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:36,317 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:36,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:36,317 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:36,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:36,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:36,377 DEBUG [Thread-823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:62055 2023-04-20 20:54:36,377 DEBUG [Thread-823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:36,378 DEBUG [Thread-829] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:62055 2023-04-20 20:54:36,378 DEBUG [Thread-829] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:36,379 DEBUG [Thread-825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:62055 2023-04-20 20:54:36,379 DEBUG [Thread-825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:36,380 DEBUG [Thread-827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:62055 2023-04-20 20:54:36,380 DEBUG [Thread-827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:36,417 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:36,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:36,445 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=766, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/bce358e01f4048a2823029e5ad1fe032 2023-04-20 20:54:36,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:36,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 584 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45368 deadline: 1682024136749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:36,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/76a29c67d00547d6912d41c542bff9e6 2023-04-20 20:54:37,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:37,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45344 deadline: 1682024137102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:37,104 DEBUG [Thread-814] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:37,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:37,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45330 deadline: 1682024137112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:37,113 DEBUG [Thread-820] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:37,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:37,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45358 deadline: 1682024137119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:37,121 DEBUG [Thread-818] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:54:37,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/3b6dea0a934a4ec3acadd833be6a14a7 2023-04-20 20:54:37,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/bce358e01f4048a2823029e5ad1fe032 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/bce358e01f4048a2823029e5ad1fe032 2023-04-20 20:54:37,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/bce358e01f4048a2823029e5ad1fe032, entries=200, sequenceid=766, filesize=32.3 K 2023-04-20 20:54:37,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/76a29c67d00547d6912d41c542bff9e6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/76a29c67d00547d6912d41c542bff9e6 2023-04-20 20:54:37,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/76a29c67d00547d6912d41c542bff9e6, entries=150, sequenceid=766, filesize=11.9 K 2023-04-20 20:54:37,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/3b6dea0a934a4ec3acadd833be6a14a7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/3b6dea0a934a4ec3acadd833be6a14a7 2023-04-20 20:54:37,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/3b6dea0a934a4ec3acadd833be6a14a7, entries=150, sequenceid=766, filesize=11.9 K 2023-04-20 20:54:37,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=26.84 KB/27480 for 3479312d5cfad95b1d49f86d4ca52095 in 1664ms, sequenceid=766, compaction requested=true 2023-04-20 20:54:37,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:37,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:37,284 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:54:37,284 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:54:37,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:37,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:37,286 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:54:37,286 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:54:37,286 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/A is initiating minor compaction (all files) 2023-04-20 20:54:37,286 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/B is initiating minor compaction (all files) 2023-04-20 20:54:37,286 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/A in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:37,286 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/B in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:37,286 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/e47f239ab5a247a88ecce8a00d45d998, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ee56db696a0d4894b3ed3d41ad926c05, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/bce358e01f4048a2823029e5ad1fe032] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=83.8 K 2023-04-20 20:54:37,286 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e6a2a12136964ad4af422e5eb900ef3e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/244374fa7f6e494ebb58c0028900e6fa, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/76a29c67d00547d6912d41c542bff9e6] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=36.3 K 2023-04-20 20:54:37,286 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e47f239ab5a247a88ecce8a00d45d998, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682024072946 2023-04-20 20:54:37,286 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e6a2a12136964ad4af422e5eb900ef3e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682024072946 2023-04-20 20:54:37,287 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ee56db696a0d4894b3ed3d41ad926c05, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1682024072982 2023-04-20 20:54:37,287 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 244374fa7f6e494ebb58c0028900e6fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1682024072982 2023-04-20 20:54:37,287 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting bce358e01f4048a2823029e5ad1fe032, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1682024074490 2023-04-20 20:54:37,287 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 76a29c67d00547d6912d41c542bff9e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1682024074490 2023-04-20 20:54:37,295 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#B#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:37,298 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#A#compaction#174 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:37,708 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/6494ff74d95c4bf8a65b9ae7688f2b23 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6494ff74d95c4bf8a65b9ae7688f2b23 2023-04-20 20:54:37,714 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/B of 3479312d5cfad95b1d49f86d4ca52095 into 6494ff74d95c4bf8a65b9ae7688f2b23(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:37,714 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:37,714 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/B, priority=13, startTime=1682024077284; duration=0sec 2023-04-20 20:54:37,714 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:37,714 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:54:37,715 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:54:37,716 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 3479312d5cfad95b1d49f86d4ca52095/C is initiating minor compaction (all files) 2023-04-20 20:54:37,716 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3479312d5cfad95b1d49f86d4ca52095/C in TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:37,716 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0d30d2e79d7a43d6b39ca4749dda0259, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b601591eee32441ab09275c17d72b503, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/3b6dea0a934a4ec3acadd833be6a14a7] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp, totalSize=36.3 K 2023-04-20 20:54:37,716 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0d30d2e79d7a43d6b39ca4749dda0259, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=649, earliestPutTs=1682024072946 2023-04-20 20:54:37,717 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b601591eee32441ab09275c17d72b503, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1682024072982 2023-04-20 20:54:37,717 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3b6dea0a934a4ec3acadd833be6a14a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1682024074490 2023-04-20 20:54:37,723 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 3479312d5cfad95b1d49f86d4ca52095#C#compaction#175 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:37,734 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/6ef2f159083c4494a0b4fa08e8e3e675 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6ef2f159083c4494a0b4fa08e8e3e675 2023-04-20 20:54:37,738 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/C of 3479312d5cfad95b1d49f86d4ca52095 into 6ef2f159083c4494a0b4fa08e8e3e675(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:37,738 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:37,738 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/C, priority=13, startTime=1682024077285; duration=0sec 2023-04-20 20:54:37,738 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:37,758 DEBUG [Thread-812] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34135972 to 127.0.0.1:62055 2023-04-20 20:54:37,758 DEBUG [Thread-812] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:38,525 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/b391793b27e7447b89064effa34ced8d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/b391793b27e7447b89064effa34ced8d 2023-04-20 20:54:38,528 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3479312d5cfad95b1d49f86d4ca52095/A of 3479312d5cfad95b1d49f86d4ca52095 into b391793b27e7447b89064effa34ced8d(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-20 20:54:38,528 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:38,529 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095., storeName=3479312d5cfad95b1d49f86d4ca52095/A, priority=13, startTime=1682024077284; duration=1sec 2023-04-20 20:54:38,529 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:39,350 DEBUG [Thread-816] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:62055 2023-04-20 20:54:39,350 DEBUG [Thread-816] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:41,746 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-20 20:54:47,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:47,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:47,169 DEBUG [Thread-814] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:62055 2023-04-20 20:54:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:47,170 DEBUG [Thread-814] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:47,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:47,208 DEBUG [Thread-818] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:62055 2023-04-20 20:54:47,208 DEBUG [Thread-818] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:47,211 DEBUG [Thread-820] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:62055 2023-04-20 20:54:47,211 DEBUG [Thread-820] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:47,211 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 20:54:47,211 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 475 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 4329 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 4262 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1815 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 5445 rows 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1807 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 5419 rows 2023-04-20 20:54:47,212 INFO [Listener at localhost/45619] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 20:54:47,212 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52274650 to 127.0.0.1:62055 2023-04-20 20:54:47,212 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:54:47,214 INFO [Listener at localhost/45619] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 20:54:47,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-20 20:54:47,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 20:54:47,219 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024087218"}]},"ts":"1682024087218"} 2023-04-20 20:54:47,220 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 20:54:47,222 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 20:54:47,222 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, UNASSIGN}] 2023-04-20 20:54:47,224 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, UNASSIGN 2023-04-20 20:54:47,224 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:47,224 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024087224"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024087224"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024087224"}]},"ts":"1682024087224"} 2023-04-20 20:54:47,225 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:54:47,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 20:54:47,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:47,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3479312d5cfad95b1d49f86d4ca52095, disabling compactions & flushes 2023-04-20 20:54:47,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:47,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 20:54:47,584 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304201b96fa734d3e472d88366685cb237d0a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201b96fa734d3e472d88366685cb237d0a 2023-04-20 20:54:47,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 20:54:47,990 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=779, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/eaa5479eaa4b4344ad54c4de351ba80b 2023-04-20 20:54:48,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 20:54:48,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/d8e61e9b60434c3b942ef2333a1f714f 2023-04-20 20:54:48,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/988d1453242a497089949c27af9bef50 2023-04-20 20:54:48,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/eaa5479eaa4b4344ad54c4de351ba80b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/eaa5479eaa4b4344ad54c4de351ba80b 2023-04-20 20:54:48,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/eaa5479eaa4b4344ad54c4de351ba80b, entries=150, sequenceid=779, filesize=25.4 K 2023-04-20 20:54:48,818 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/d8e61e9b60434c3b942ef2333a1f714f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/d8e61e9b60434c3b942ef2333a1f714f 2023-04-20 20:54:48,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/d8e61e9b60434c3b942ef2333a1f714f, entries=150, sequenceid=779, filesize=11.9 K 2023-04-20 20:54:48,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/988d1453242a497089949c27af9bef50 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/988d1453242a497089949c27af9bef50 2023-04-20 20:54:48,826 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/988d1453242a497089949c27af9bef50, entries=150, sequenceid=779, filesize=11.9 K 2023-04-20 20:54:48,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for 3479312d5cfad95b1d49f86d4ca52095 in 1657ms, sequenceid=779, compaction requested=false 2023-04-20 20:54:48,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:48,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. after waiting 0 ms 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:48,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3479312d5cfad95b1d49f86d4ca52095 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=A 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=B 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3479312d5cfad95b1d49f86d4ca52095, store=C 2023-04-20 20:54:48,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:49,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304206814934349574a85a9dbf98131c0ed86 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206814934349574a85a9dbf98131c0ed86 2023-04-20 20:54:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 20:54:49,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=784, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/9405e936b3874c99b05e9377a891f219 2023-04-20 20:54:50,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/03de69a6ee264d3ea91fed86349d5293 2023-04-20 20:54:50,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/df4c6b4a5ef94e508f88d9f60f486e5a 2023-04-20 20:54:50,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/A/9405e936b3874c99b05e9377a891f219 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9405e936b3874c99b05e9377a891f219 2023-04-20 20:54:50,477 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9405e936b3874c99b05e9377a891f219, entries=100, sequenceid=784, filesize=18.6 K 2023-04-20 20:54:50,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/B/03de69a6ee264d3ea91fed86349d5293 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/03de69a6ee264d3ea91fed86349d5293 2023-04-20 20:54:50,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/03de69a6ee264d3ea91fed86349d5293, entries=100, sequenceid=784, filesize=9.5 K 2023-04-20 20:54:50,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/.tmp/C/df4c6b4a5ef94e508f88d9f60f486e5a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/df4c6b4a5ef94e508f88d9f60f486e5a 2023-04-20 20:54:50,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/df4c6b4a5ef94e508f88d9f60f486e5a, entries=100, sequenceid=784, filesize=9.5 K 2023-04-20 20:54:50,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 3479312d5cfad95b1d49f86d4ca52095 in 1659ms, sequenceid=784, compaction requested=true 2023-04-20 20:54:50,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f903aa554894459f8d49a43ce83bcdfc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/4e82ca4c119f4cc99feedd125e7b616f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ed9337049e2c48fb86a1ed785ce4c5bd, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/a43965da9d2448e990bc3b7977f1e752, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/312f95734f3743449ec77d878b454124, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/48f5ff774dc44295bd0ed05342a8be87, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/7732968ce84a43b191134e55ea41cd4e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9859022c51ce4140908e6fb5c59726d5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f4e74f67cc874643ab39c4c75618ad7d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/98206da4305f4dcab156616e53718e5c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/11cd1e432754489e88346e75906073d6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f7d40973f76e4b979ebe48be9a92f42b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/499387492fd94834bbee159de9164fa8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f59c029190114a8ca851928c61d3d786, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/89a3500afdb54e9ebb7c77ce83cd518d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/2cea0b8e95d040ceb4ba71f50d37c8ee, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/0f945371ea03493d881987d029749e7a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/6de72be12e994a19a38107d3cf9f6494, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/e47f239ab5a247a88ecce8a00d45d998, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ee56db696a0d4894b3ed3d41ad926c05, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/bce358e01f4048a2823029e5ad1fe032] to archive 2023-04-20 20:54:50,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:54:50,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f903aa554894459f8d49a43ce83bcdfc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f903aa554894459f8d49a43ce83bcdfc 2023-04-20 20:54:50,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/4e82ca4c119f4cc99feedd125e7b616f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/4e82ca4c119f4cc99feedd125e7b616f 2023-04-20 20:54:50,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ed9337049e2c48fb86a1ed785ce4c5bd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ed9337049e2c48fb86a1ed785ce4c5bd 2023-04-20 20:54:50,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/a43965da9d2448e990bc3b7977f1e752 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/a43965da9d2448e990bc3b7977f1e752 2023-04-20 20:54:50,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/312f95734f3743449ec77d878b454124 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/312f95734f3743449ec77d878b454124 2023-04-20 20:54:50,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/48f5ff774dc44295bd0ed05342a8be87 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/48f5ff774dc44295bd0ed05342a8be87 2023-04-20 20:54:50,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/7732968ce84a43b191134e55ea41cd4e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/7732968ce84a43b191134e55ea41cd4e 2023-04-20 20:54:50,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9859022c51ce4140908e6fb5c59726d5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9859022c51ce4140908e6fb5c59726d5 2023-04-20 20:54:50,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f4e74f67cc874643ab39c4c75618ad7d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f4e74f67cc874643ab39c4c75618ad7d 2023-04-20 20:54:50,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/98206da4305f4dcab156616e53718e5c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/98206da4305f4dcab156616e53718e5c 2023-04-20 20:54:50,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/11cd1e432754489e88346e75906073d6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/11cd1e432754489e88346e75906073d6 2023-04-20 20:54:50,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f7d40973f76e4b979ebe48be9a92f42b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f7d40973f76e4b979ebe48be9a92f42b 2023-04-20 20:54:50,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/499387492fd94834bbee159de9164fa8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/499387492fd94834bbee159de9164fa8 2023-04-20 20:54:50,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f59c029190114a8ca851928c61d3d786 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/f59c029190114a8ca851928c61d3d786 2023-04-20 20:54:50,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/89a3500afdb54e9ebb7c77ce83cd518d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/89a3500afdb54e9ebb7c77ce83cd518d 2023-04-20 20:54:50,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/2cea0b8e95d040ceb4ba71f50d37c8ee to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/2cea0b8e95d040ceb4ba71f50d37c8ee 2023-04-20 20:54:50,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/0f945371ea03493d881987d029749e7a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/0f945371ea03493d881987d029749e7a 2023-04-20 20:54:50,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/6de72be12e994a19a38107d3cf9f6494 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/6de72be12e994a19a38107d3cf9f6494 2023-04-20 20:54:50,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/e47f239ab5a247a88ecce8a00d45d998 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/e47f239ab5a247a88ecce8a00d45d998 2023-04-20 20:54:50,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ee56db696a0d4894b3ed3d41ad926c05 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/ee56db696a0d4894b3ed3d41ad926c05 2023-04-20 20:54:50,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/bce358e01f4048a2823029e5ad1fe032 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/bce358e01f4048a2823029e5ad1fe032 2023-04-20 20:54:50,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/c0584a32196e4c83934603348c6ce1c3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e32e123686504c5da9d11b6b9c72d7e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fde71672592245769e38a0470e6b80c4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/1d460e0c12a84558b152f4235c92a7ae, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6395b536ddba4ccd99cfd6551552809e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e08845360406493e8060a0a41db7948a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/4c8b41f812b94e819336d15c395a2c99, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aefeef38560f44beb30642e509f33acf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aa50e3d2abf7463292726aacc29d8977, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/813f51cb5f014cfdb1e8fcee0504ebec, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fefb875567954c938d9d2cc26bebd899, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/beb7c14dcd6e41148200cd7e4eba403d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/315bfd5d894743ddac76381d3d77ccb5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/698fc42038de4c1f982b88062d71de61, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/783d78e718724f45b1dacb5990df5d9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/7db67099cee443daba5abb5017a4e409, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6bfdb2b9915d46feac6a61554e88483d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e6a2a12136964ad4af422e5eb900ef3e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/61433609442a4775b9a02326c7531c35, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/244374fa7f6e494ebb58c0028900e6fa, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/76a29c67d00547d6912d41c542bff9e6] to archive 2023-04-20 20:54:50,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:54:50,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/c0584a32196e4c83934603348c6ce1c3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/c0584a32196e4c83934603348c6ce1c3 2023-04-20 20:54:50,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e32e123686504c5da9d11b6b9c72d7e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e32e123686504c5da9d11b6b9c72d7e7 2023-04-20 20:54:50,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fde71672592245769e38a0470e6b80c4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fde71672592245769e38a0470e6b80c4 2023-04-20 20:54:50,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/1d460e0c12a84558b152f4235c92a7ae to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/1d460e0c12a84558b152f4235c92a7ae 2023-04-20 20:54:50,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6395b536ddba4ccd99cfd6551552809e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6395b536ddba4ccd99cfd6551552809e 2023-04-20 20:54:50,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e08845360406493e8060a0a41db7948a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e08845360406493e8060a0a41db7948a 2023-04-20 20:54:50,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/4c8b41f812b94e819336d15c395a2c99 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/4c8b41f812b94e819336d15c395a2c99 2023-04-20 20:54:50,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aefeef38560f44beb30642e509f33acf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aefeef38560f44beb30642e509f33acf 2023-04-20 20:54:50,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aa50e3d2abf7463292726aacc29d8977 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/aa50e3d2abf7463292726aacc29d8977 2023-04-20 20:54:50,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/813f51cb5f014cfdb1e8fcee0504ebec to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/813f51cb5f014cfdb1e8fcee0504ebec 2023-04-20 20:54:50,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fefb875567954c938d9d2cc26bebd899 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/fefb875567954c938d9d2cc26bebd899 2023-04-20 20:54:50,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/beb7c14dcd6e41148200cd7e4eba403d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/beb7c14dcd6e41148200cd7e4eba403d 2023-04-20 20:54:50,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/315bfd5d894743ddac76381d3d77ccb5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/315bfd5d894743ddac76381d3d77ccb5 2023-04-20 20:54:50,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/698fc42038de4c1f982b88062d71de61 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/698fc42038de4c1f982b88062d71de61 2023-04-20 20:54:50,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/783d78e718724f45b1dacb5990df5d9d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/783d78e718724f45b1dacb5990df5d9d 2023-04-20 20:54:50,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/7db67099cee443daba5abb5017a4e409 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/7db67099cee443daba5abb5017a4e409 2023-04-20 20:54:50,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6bfdb2b9915d46feac6a61554e88483d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6bfdb2b9915d46feac6a61554e88483d 2023-04-20 20:54:50,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e6a2a12136964ad4af422e5eb900ef3e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/e6a2a12136964ad4af422e5eb900ef3e 2023-04-20 20:54:50,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/61433609442a4775b9a02326c7531c35 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/61433609442a4775b9a02326c7531c35 2023-04-20 20:54:50,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/244374fa7f6e494ebb58c0028900e6fa to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/244374fa7f6e494ebb58c0028900e6fa 2023-04-20 20:54:50,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/76a29c67d00547d6912d41c542bff9e6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/76a29c67d00547d6912d41c542bff9e6 2023-04-20 20:54:50,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/12c11b2282d84252ab11dcc66f739038, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/f2b60286cab043df988e0639150c6532, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/208159da14754cfe8e04d6a8027eb615, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/4e0ea1b371b54d2d9bb8288ae3955bde, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/ecfe9f645563426e9e0af64223538819, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0347f183a1a64de5bfeedcdf4a7af0ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/45685b5104774c4fb301a817b59d396a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8e2c3d2e3f7749a18406f418fea92e45, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7764c971434b43778f4f443cd7b4d3dc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/927bbbcbe87d43d88b44f2d7130329f2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7f9a51a3f98744dcbced1f7f86c6c51e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/5f9467b5842545f4a62691f02a4abc0f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6410650d91c0421bb1aa2f9414e35388, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/c9bc35ee17d24c81b219ccc55ef24d19, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8a4dc42800c049ee80a7998b1a2448ef, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7905f96943474d95b22f8f393a48d987, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b941c030e4784d269f349f484cfcd3bc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0d30d2e79d7a43d6b39ca4749dda0259, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/55291c51494e4d388d05435911fb5fed, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b601591eee32441ab09275c17d72b503, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/3b6dea0a934a4ec3acadd833be6a14a7] to archive 2023-04-20 20:54:50,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:54:50,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/12c11b2282d84252ab11dcc66f739038 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/12c11b2282d84252ab11dcc66f739038 2023-04-20 20:54:50,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/f2b60286cab043df988e0639150c6532 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/f2b60286cab043df988e0639150c6532 2023-04-20 20:54:50,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/208159da14754cfe8e04d6a8027eb615 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/208159da14754cfe8e04d6a8027eb615 2023-04-20 20:54:50,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/4e0ea1b371b54d2d9bb8288ae3955bde to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/4e0ea1b371b54d2d9bb8288ae3955bde 2023-04-20 20:54:50,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/ecfe9f645563426e9e0af64223538819 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/ecfe9f645563426e9e0af64223538819 2023-04-20 20:54:50,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0347f183a1a64de5bfeedcdf4a7af0ad to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0347f183a1a64de5bfeedcdf4a7af0ad 2023-04-20 20:54:50,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/45685b5104774c4fb301a817b59d396a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/45685b5104774c4fb301a817b59d396a 2023-04-20 20:54:50,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8e2c3d2e3f7749a18406f418fea92e45 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8e2c3d2e3f7749a18406f418fea92e45 2023-04-20 20:54:50,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7764c971434b43778f4f443cd7b4d3dc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7764c971434b43778f4f443cd7b4d3dc 2023-04-20 20:54:50,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/927bbbcbe87d43d88b44f2d7130329f2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/927bbbcbe87d43d88b44f2d7130329f2 2023-04-20 20:54:50,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7f9a51a3f98744dcbced1f7f86c6c51e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7f9a51a3f98744dcbced1f7f86c6c51e 2023-04-20 20:54:50,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/5f9467b5842545f4a62691f02a4abc0f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/5f9467b5842545f4a62691f02a4abc0f 2023-04-20 20:54:50,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6410650d91c0421bb1aa2f9414e35388 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6410650d91c0421bb1aa2f9414e35388 2023-04-20 20:54:50,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/c9bc35ee17d24c81b219ccc55ef24d19 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/c9bc35ee17d24c81b219ccc55ef24d19 2023-04-20 20:54:50,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8a4dc42800c049ee80a7998b1a2448ef to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/8a4dc42800c049ee80a7998b1a2448ef 2023-04-20 20:54:50,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7905f96943474d95b22f8f393a48d987 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/7905f96943474d95b22f8f393a48d987 2023-04-20 20:54:50,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b941c030e4784d269f349f484cfcd3bc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b941c030e4784d269f349f484cfcd3bc 2023-04-20 20:54:50,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0d30d2e79d7a43d6b39ca4749dda0259 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/0d30d2e79d7a43d6b39ca4749dda0259 2023-04-20 20:54:50,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/55291c51494e4d388d05435911fb5fed to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/55291c51494e4d388d05435911fb5fed 2023-04-20 20:54:50,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b601591eee32441ab09275c17d72b503 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/b601591eee32441ab09275c17d72b503 2023-04-20 20:54:50,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/3b6dea0a934a4ec3acadd833be6a14a7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/3b6dea0a934a4ec3acadd833be6a14a7 2023-04-20 20:54:50,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/recovered.edits/787.seqid, newMaxSeqId=787, maxSeqId=4 2023-04-20 20:54:50,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095. 2023-04-20 20:54:50,579 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3479312d5cfad95b1d49f86d4ca52095: 2023-04-20 20:54:50,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:50,581 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=3479312d5cfad95b1d49f86d4ca52095, regionState=CLOSED 2023-04-20 20:54:50,581 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024090581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024090581"}]},"ts":"1682024090581"} 2023-04-20 20:54:50,583 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-20 20:54:50,583 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 3479312d5cfad95b1d49f86d4ca52095, server=jenkins-hbase4.apache.org,39223,1682024020002 in 3.3570 sec 2023-04-20 20:54:50,584 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-20 20:54:50,584 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3479312d5cfad95b1d49f86d4ca52095, UNASSIGN in 3.3610 sec 2023-04-20 20:54:50,585 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024090585"}]},"ts":"1682024090585"} 2023-04-20 20:54:50,585 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 20:54:50,587 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 20:54:50,588 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3720 sec 2023-04-20 20:54:51,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 20:54:51,324 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-20 20:54:51,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-20 20:54:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:51,327 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:51,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-20 20:54:51,327 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:51,332 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:51,335 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/recovered.edits] 2023-04-20 20:54:51,338 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9405e936b3874c99b05e9377a891f219 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/9405e936b3874c99b05e9377a891f219 2023-04-20 20:54:51,339 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/b391793b27e7447b89064effa34ced8d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/b391793b27e7447b89064effa34ced8d 2023-04-20 20:54:51,341 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/eaa5479eaa4b4344ad54c4de351ba80b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/A/eaa5479eaa4b4344ad54c4de351ba80b 2023-04-20 20:54:51,343 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/03de69a6ee264d3ea91fed86349d5293 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/03de69a6ee264d3ea91fed86349d5293 2023-04-20 20:54:51,344 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6494ff74d95c4bf8a65b9ae7688f2b23 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/6494ff74d95c4bf8a65b9ae7688f2b23 2023-04-20 20:54:51,346 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/d8e61e9b60434c3b942ef2333a1f714f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/B/d8e61e9b60434c3b942ef2333a1f714f 2023-04-20 20:54:51,348 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6ef2f159083c4494a0b4fa08e8e3e675 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/6ef2f159083c4494a0b4fa08e8e3e675 2023-04-20 20:54:51,350 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/988d1453242a497089949c27af9bef50 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/988d1453242a497089949c27af9bef50 2023-04-20 20:54:51,351 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/df4c6b4a5ef94e508f88d9f60f486e5a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/C/df4c6b4a5ef94e508f88d9f60f486e5a 2023-04-20 20:54:51,354 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/recovered.edits/787.seqid to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095/recovered.edits/787.seqid 2023-04-20 20:54:51,355 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/3479312d5cfad95b1d49f86d4ca52095 2023-04-20 20:54:51,355 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:54:51,355 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 20:54:51,356 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-20 20:54:51,359 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420017b21da29524a0dbaa8884dca3b1819 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420017b21da29524a0dbaa8884dca3b1819 2023-04-20 20:54:51,360 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200369ddc96a9a4fb0ae42c59d9313dd53 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200369ddc96a9a4fb0ae42c59d9313dd53 2023-04-20 20:54:51,361 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201b96fa734d3e472d88366685cb237d0a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201b96fa734d3e472d88366685cb237d0a 2023-04-20 20:54:51,362 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202753947fe9614bb6b6b1e74a22afb159 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202753947fe9614bb6b6b1e74a22afb159 2023-04-20 20:54:51,363 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202802a7cb6fb24dd7a35630ddf1dbf043 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202802a7cb6fb24dd7a35630ddf1dbf043 2023-04-20 20:54:51,364 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202ce12673a29f4cfb9555ddf460afe9a2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202ce12673a29f4cfb9555ddf460afe9a2 2023-04-20 20:54:51,365 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203eb06a41d75c49f7985e988ac16d2694 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203eb06a41d75c49f7985e988ac16d2694 2023-04-20 20:54:51,366 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205b7188509a2e491eb4fef03008d424af to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205b7188509a2e491eb4fef03008d424af 2023-04-20 20:54:51,367 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042061e0c31f4982415f878ff00a5d79594e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042061e0c31f4982415f878ff00a5d79594e 2023-04-20 20:54:51,368 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206814934349574a85a9dbf98131c0ed86 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206814934349574a85a9dbf98131c0ed86 2023-04-20 20:54:51,369 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042086078f9f6f3443cf822a91d84b597fdd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042086078f9f6f3443cf822a91d84b597fdd 2023-04-20 20:54:51,370 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a0261cdef95b49619cd329f24b90818b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a0261cdef95b49619cd329f24b90818b 2023-04-20 20:54:51,371 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a26ee3a98ee5483699d1061805072be1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a26ee3a98ee5483699d1061805072be1 2023-04-20 20:54:51,372 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a621346717214f6b8a41da30c4fed8cf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a621346717214f6b8a41da30c4fed8cf 2023-04-20 20:54:51,373 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bada722ca70d47ed9b2cc4930778f0da to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bada722ca70d47ed9b2cc4930778f0da 2023-04-20 20:54:51,375 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bcb1c2aa94d9475ba880b2db00a87842 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bcb1c2aa94d9475ba880b2db00a87842 2023-04-20 20:54:51,376 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420be5fcf436ecd4e4fa4b79a23be6c52b3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420be5fcf436ecd4e4fa4b79a23be6c52b3 2023-04-20 20:54:51,377 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420dce17f81ded64f00af472fcc62526069 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420dce17f81ded64f00af472fcc62526069 2023-04-20 20:54:51,378 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e6dce45d3fa64823aace075348537a4c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e6dce45d3fa64823aace075348537a4c 2023-04-20 20:54:51,379 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420f5e81aba03b1484499a727da2fc003d3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420f5e81aba03b1484499a727da2fc003d3 2023-04-20 20:54:51,379 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 20:54:51,381 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:51,383 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 20:54:51,385 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 20:54:51,386 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:51,386 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 20:54:51,386 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682024091386"}]},"ts":"9223372036854775807"} 2023-04-20 20:54:51,388 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 20:54:51,388 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3479312d5cfad95b1d49f86d4ca52095, NAME => 'TestAcidGuarantees,,1682024053236.3479312d5cfad95b1d49f86d4ca52095.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 20:54:51,388 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 20:54:51,388 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682024091388"}]},"ts":"9223372036854775807"} 2023-04-20 20:54:51,389 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 20:54:51,391 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:51,392 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-04-20 20:54:51,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-20 20:54:51,428 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-20 20:54:51,440 INFO [Listener at localhost/45619] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=225 (was 232), OpenFileDescriptor=489 (was 560), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=436 (was 487), ProcessCount=169 (was 169), AvailableMemoryMB=8704 (was 8850) 2023-04-20 20:54:51,451 INFO [Listener at localhost/45619] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=225, OpenFileDescriptor=489, MaxFileDescriptor=60000, SystemLoadAverage=436, ProcessCount=169, AvailableMemoryMB=8703 2023-04-20 20:54:51,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:54:51,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:54:51,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 20:54:51,456 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 20:54:51,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-20 20:54:51,456 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 20:54:51,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 20:54:51,457 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:51,458 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3 empty. 2023-04-20 20:54:51,458 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:51,458 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:54:51,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 20:54:51,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 20:54:51,866 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:54:51,867 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bd7cf796b1328b6fd145cda0a337b2e3, NAME => 'TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp 2023-04-20 20:54:52,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 20:54:52,273 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:54:52,273 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bd7cf796b1328b6fd145cda0a337b2e3, disabling compactions & flushes 2023-04-20 20:54:52,273 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,273 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,273 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. after waiting 0 ms 2023-04-20 20:54:52,273 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,273 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,273 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:52,276 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 20:54:52,276 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024092276"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024092276"}]},"ts":"1682024092276"} 2023-04-20 20:54:52,277 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 20:54:52,278 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 20:54:52,278 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024092278"}]},"ts":"1682024092278"} 2023-04-20 20:54:52,279 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 20:54:52,283 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bd7cf796b1328b6fd145cda0a337b2e3, ASSIGN}] 2023-04-20 20:54:52,284 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bd7cf796b1328b6fd145cda0a337b2e3, ASSIGN 2023-04-20 20:54:52,285 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bd7cf796b1328b6fd145cda0a337b2e3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=false 2023-04-20 20:54:52,436 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=bd7cf796b1328b6fd145cda0a337b2e3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:52,436 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024092436"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024092436"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024092436"}]},"ts":"1682024092436"} 2023-04-20 20:54:52,437 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:54:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 20:54:52,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bd7cf796b1328b6fd145cda0a337b2e3, NAME => 'TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:54:52,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:54:52,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,596 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,597 DEBUG [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A 2023-04-20 20:54:52,597 DEBUG [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A 2023-04-20 20:54:52,597 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:52,598 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd7cf796b1328b6fd145cda0a337b2e3 columnFamilyName A 2023-04-20 20:54:52,598 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.HStore(310): Store=bd7cf796b1328b6fd145cda0a337b2e3/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:52,598 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,599 DEBUG [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B 2023-04-20 20:54:52,600 DEBUG [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B 2023-04-20 20:54:52,600 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:52,600 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd7cf796b1328b6fd145cda0a337b2e3 columnFamilyName B 2023-04-20 20:54:52,600 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.HStore(310): Store=bd7cf796b1328b6fd145cda0a337b2e3/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:52,600 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,601 DEBUG [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C 2023-04-20 20:54:52,601 DEBUG [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C 2023-04-20 20:54:52,601 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:54:52,602 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd7cf796b1328b6fd145cda0a337b2e3 columnFamilyName C 2023-04-20 20:54:52,602 INFO [StoreOpener-bd7cf796b1328b6fd145cda0a337b2e3-1] regionserver.HStore(310): Store=bd7cf796b1328b6fd145cda0a337b2e3/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:54:52,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:54:52,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:52,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:54:52,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bd7cf796b1328b6fd145cda0a337b2e3; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65370110, jitterRate=-0.025909453630447388}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:54:52,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:52,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., pid=30, masterSystemTime=1682024092588 2023-04-20 20:54:52,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:52,611 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=bd7cf796b1328b6fd145cda0a337b2e3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:52,611 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024092611"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024092611"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024092611"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024092611"}]},"ts":"1682024092611"} 2023-04-20 20:54:52,613 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-20 20:54:52,613 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 in 175 msec 2023-04-20 20:54:52,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-20 20:54:52,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bd7cf796b1328b6fd145cda0a337b2e3, ASSIGN in 331 msec 2023-04-20 20:54:52,615 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 20:54:52,615 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024092615"}]},"ts":"1682024092615"} 2023-04-20 20:54:52,616 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 20:54:52,619 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 20:54:52,620 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-04-20 20:54:53,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 20:54:53,561 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-20 20:54:53,562 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,567 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,568 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,569 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57974, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,570 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 20:54:53,573 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 20:54:53,574 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,579 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,580 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,585 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,585 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,589 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,589 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,593 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,593 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,597 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,597 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,601 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,601 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,606 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,606 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,610 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,610 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,614 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,615 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:54:53,620 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:54:53,625 DEBUG [hconnection-0x55367947-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,625 DEBUG [hconnection-0x292a585-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,625 DEBUG [hconnection-0x17abd6b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:53,627 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57984, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,628 DEBUG [hconnection-0x6a1ded95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,628 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,629 DEBUG [hconnection-0x2b59d60a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,629 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,628 DEBUG [hconnection-0x28b936cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,628 DEBUG [hconnection-0x208d05ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,630 DEBUG [hconnection-0x4843e6b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,630 DEBUG [hconnection-0x5658fb6f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,633 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,633 DEBUG [hconnection-0x16674b36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:54:53,633 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,636 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,636 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,637 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,637 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,637 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:54:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:53,639 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:53,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:53,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:53,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:53,642 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,642 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:53,646 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:53,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:53,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:53,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:53,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:53,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:53,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:53,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:53,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:53,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:53,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:53,652 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:53,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024093661, max timestamp=1682024093668Num uniques 859; 2023-04-20 20:54:53,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:53,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024093661, max timestamp=1682024093668Num uniques 859; 2023-04-20 20:54:53,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024093661, max timestamp=1682024093668Num uniques 859; 2023-04-20 20:54:53,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:53,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:54:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:53,673 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:53,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:53,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:53,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc88998[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:53,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:53,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,679 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:53,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:53,683 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024153682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:53,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024153681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,686 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:53,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:53,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:53,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:53,686 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:53,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024153684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:53,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024153686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024153686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:53,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:53,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:53,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:53,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:53,694 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:53,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:53,694 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:53,701 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/ccd656976ca744aeb3a4361fda939708 2023-04-20 20:54:53,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024153787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024153787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024153788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024153791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024153791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,794 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:53,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:53,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:53,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:53,798 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:53,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:53,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:53,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:53,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,800 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:53,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:53,801 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,801 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:53,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:53,802 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:53,802 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:53,802 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:53,802 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:53,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:53,810 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5029cd6e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:53,812 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:53,812 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:53,814 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,814 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:53,815 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:53,818 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:53,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:53,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:53,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:53,818 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:53,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,820 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,820 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:53,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:53,820 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:53,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:53,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:53,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:53,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:53,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:53,826 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:53,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:53,826 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:53,826 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:53,926 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:53,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:53,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:53,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:53,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:53,930 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:53,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:53,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:53,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:53,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:53,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:53,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:53,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:53,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:53,934 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:53,934 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:53,934 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:53,934 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:53,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:53,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ddb457c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:53,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:53,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,939 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:53,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:53,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,941 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,942 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:53,942 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:53,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:53,944 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:53,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:53,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:53,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:53,944 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:53,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:53,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,946 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:53,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,947 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:53,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:53,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:53,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:53,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:53,950 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:53,950 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:53,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:53,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:53,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024153990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024153990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024153992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024153993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:53,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024153993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,051 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:54,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:54,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:54,054 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:54,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:54,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:54,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:54,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,056 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:54,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:54,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:54,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:54,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:54,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:54,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:54,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:54,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:54,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:54,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:54,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:54,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:54,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:54,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:54,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:54,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:54,062 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,062 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdbbc76[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:54,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:54,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:54,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:54,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:54,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:54,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:54,070 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:54,070 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:54,070 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:54,070 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:54,070 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:54,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:54,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:54,072 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:54,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:54,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:54,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:54,081 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:54,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:54,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:54,081 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:54,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:54,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:54,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/b31cf195360f42b4a5b84e348d054ca3 2023-04-20 20:54:54,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/5f6177759224493784828f047a93654b 2023-04-20 20:54:54,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/ccd656976ca744aeb3a4361fda939708 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ccd656976ca744aeb3a4361fda939708 2023-04-20 20:54:54,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ccd656976ca744aeb3a4361fda939708, entries=150, sequenceid=12, filesize=11.6 K 2023-04-20 20:54:54,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/b31cf195360f42b4a5b84e348d054ca3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b31cf195360f42b4a5b84e348d054ca3 2023-04-20 20:54:54,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b31cf195360f42b4a5b84e348d054ca3, entries=150, sequenceid=12, filesize=11.6 K 2023-04-20 20:54:54,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/5f6177759224493784828f047a93654b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5f6177759224493784828f047a93654b 2023-04-20 20:54:54,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5f6177759224493784828f047a93654b, entries=150, sequenceid=12, filesize=11.6 K 2023-04-20 20:54:54,151 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for bd7cf796b1328b6fd145cda0a337b2e3 in 505ms, sequenceid=12, compaction requested=false 2023-04-20 20:54:54,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:54,182 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:54,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:54,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:54,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:54,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:54,185 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:54,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:54,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:54,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:54,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,187 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:54,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:54,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:54,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:54,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:54,189 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:54,189 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:54,189 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-20 20:54:54,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:54,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:54,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:54,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:54,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:54,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:54,198 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/664490db0a564cc691434a40f48e22bb 2023-04-20 20:54:54,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:54,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:54,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024154298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024154299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024154300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024154300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024154301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024154401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024154403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024154403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024154403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024154404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024154604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024154606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024154607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024154607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024154608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,612 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/2ad257c8acba462e8cf63f9762acb162 2023-04-20 20:54:54,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024154907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024154909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024154910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024154911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:54,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:54,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024154911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,027 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/9281a5a166cc416eb26b8ee47e153b63 2023-04-20 20:54:55,031 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/664490db0a564cc691434a40f48e22bb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/664490db0a564cc691434a40f48e22bb 2023-04-20 20:54:55,037 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/664490db0a564cc691434a40f48e22bb, entries=150, sequenceid=47, filesize=11.6 K 2023-04-20 20:54:55,038 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/2ad257c8acba462e8cf63f9762acb162 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/2ad257c8acba462e8cf63f9762acb162 2023-04-20 20:54:55,042 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/2ad257c8acba462e8cf63f9762acb162, entries=150, sequenceid=47, filesize=11.6 K 2023-04-20 20:54:55,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/9281a5a166cc416eb26b8ee47e153b63 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/9281a5a166cc416eb26b8ee47e153b63 2023-04-20 20:54:55,047 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/9281a5a166cc416eb26b8ee47e153b63, entries=150, sequenceid=47, filesize=11.6 K 2023-04-20 20:54:55,048 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for bd7cf796b1328b6fd145cda0a337b2e3 in 859ms, sequenceid=47, compaction requested=false 2023-04-20 20:54:55,049 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:55,049 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c07c3c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:55,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:55,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,059 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:55,060 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,060 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:55,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:55,064 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:55,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:55,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:55,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:55,064 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:55,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:55,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:55,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:55,069 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:55,072 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:55,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:55,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:55,072 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:55,072 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:55,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:55,072 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:55,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:55,173 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:55,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:55,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:55,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:55,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:55,177 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:55,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:55,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:55,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:55,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:55,180 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:55,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:55,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:55,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:55,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:55,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:55,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:55,182 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:55,182 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:55,182 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:54:55,183 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:55,183 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:55,183 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:55,183 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:55,183 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:55,183 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:55,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:55,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:55,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024155425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024155426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024095422, max timestamp=1682024095426Num uniques 859; 2023-04-20 20:54:55,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024095422, max timestamp=1682024095429Num uniques 859; 2023-04-20 20:54:55,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:55,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024095422, max timestamp=1682024095426Num uniques 859; 2023-04-20 20:54:55,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:55,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:55,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024155466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024155467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024155468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024155551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024155551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024155571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024155572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024155572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,602 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/2e7d97b6ab564cd881972c3287bd0d10 2023-04-20 20:54:55,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024155765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024155765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024155773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024155775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:55,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:55,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024155776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,022 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/12913d44284b41c3a71f92ee3ceb00b5 2023-04-20 20:54:56,033 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f5593dfd3adc4b59ab07cc8c33418e14 2023-04-20 20:54:56,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/2e7d97b6ab564cd881972c3287bd0d10 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2e7d97b6ab564cd881972c3287bd0d10 2023-04-20 20:54:56,047 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2e7d97b6ab564cd881972c3287bd0d10, entries=100, sequenceid=57, filesize=9.3 K 2023-04-20 20:54:56,048 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/12913d44284b41c3a71f92ee3ceb00b5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/12913d44284b41c3a71f92ee3ceb00b5 2023-04-20 20:54:56,053 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/12913d44284b41c3a71f92ee3ceb00b5, entries=100, sequenceid=57, filesize=9.3 K 2023-04-20 20:54:56,054 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f5593dfd3adc4b59ab07cc8c33418e14 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f5593dfd3adc4b59ab07cc8c33418e14 2023-04-20 20:54:56,062 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f5593dfd3adc4b59ab07cc8c33418e14, entries=100, sequenceid=57, filesize=9.3 K 2023-04-20 20:54:56,063 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bd7cf796b1328b6fd145cda0a337b2e3 in 880ms, sequenceid=57, compaction requested=true 2023-04-20 20:54:56,063 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:56,063 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:56,064 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d9c9d97[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:56,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:56,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,068 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:56,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:56,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-20 20:54:56,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:54:56,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:56,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:54:56,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:56,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:56,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:56,073 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:56,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:56,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:56,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:56,073 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:56,077 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,077 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,077 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,079 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:56,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,083 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:56,083 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:56,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024156079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024156080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024156080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024156081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/a214adce74f14f55aceaa1abfc9351f6 2023-04-20 20:54:56,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024156082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/9c54dc252a274f37a822f85fb39033bb 2023-04-20 20:54:56,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/70beb237988346d79cf27097c00e1d26 2023-04-20 20:54:56,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/a214adce74f14f55aceaa1abfc9351f6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a214adce74f14f55aceaa1abfc9351f6 2023-04-20 20:54:56,113 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 20:54:56,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a214adce74f14f55aceaa1abfc9351f6, entries=150, sequenceid=95, filesize=11.6 K 2023-04-20 20:54:56,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/9c54dc252a274f37a822f85fb39033bb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9c54dc252a274f37a822f85fb39033bb 2023-04-20 20:54:56,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9c54dc252a274f37a822f85fb39033bb, entries=150, sequenceid=95, filesize=11.6 K 2023-04-20 20:54:56,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/70beb237988346d79cf27097c00e1d26 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/70beb237988346d79cf27097c00e1d26 2023-04-20 20:54:56,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/70beb237988346d79cf27097c00e1d26, entries=150, sequenceid=95, filesize=11.6 K 2023-04-20 20:54:56,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for bd7cf796b1328b6fd145cda0a337b2e3 in 79ms, sequenceid=95, compaction requested=true 2023-04-20 20:54:56,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:56,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:56,149 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:56,149 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:56,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:56,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:56,151 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:56,151 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:56,151 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:54:56,151 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:54:56,151 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,151 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,151 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b31cf195360f42b4a5b84e348d054ca3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/2ad257c8acba462e8cf63f9762acb162, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/12913d44284b41c3a71f92ee3ceb00b5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9c54dc252a274f37a822f85fb39033bb] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=44.1 K 2023-04-20 20:54:56,151 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ccd656976ca744aeb3a4361fda939708, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/664490db0a564cc691434a40f48e22bb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2e7d97b6ab564cd881972c3287bd0d10, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a214adce74f14f55aceaa1abfc9351f6] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=44.1 K 2023-04-20 20:54:56,152 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b31cf195360f42b4a5b84e348d054ca3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682024093643 2023-04-20 20:54:56,152 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ccd656976ca744aeb3a4361fda939708, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682024093643 2023-04-20 20:54:56,152 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2ad257c8acba462e8cf63f9762acb162, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024093683 2023-04-20 20:54:56,152 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 664490db0a564cc691434a40f48e22bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024093683 2023-04-20 20:54:56,153 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 12913d44284b41c3a71f92ee3ceb00b5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682024094298 2023-04-20 20:54:56,153 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2e7d97b6ab564cd881972c3287bd0d10, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682024094298 2023-04-20 20:54:56,153 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9c54dc252a274f37a822f85fb39033bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682024095429 2023-04-20 20:54:56,153 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a214adce74f14f55aceaa1abfc9351f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682024095429 2023-04-20 20:54:56,164 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#194 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:56,168 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#195 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:56,183 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:56,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:56,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:56,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:56,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 20:54:56,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:56,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:56,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:56,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:56,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:56,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:56,202 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:56,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:56,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:56,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:56,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:56,217 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,217 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:56,217 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:56,217 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:56,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:56,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:56,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:56,219 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,219 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,219 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,219 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,222 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/31738304aa254b0cbdb532b61edf184d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/31738304aa254b0cbdb532b61edf184d 2023-04-20 20:54:56,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:56,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a3bd5fd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:56,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:56,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,224 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:56,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:56,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:56,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:56,229 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:56,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:56,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:56,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:56,229 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:56,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:56,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,235 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:56,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,235 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,235 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:56,235 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:56,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,235 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:56,238 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into 31738304aa254b0cbdb532b61edf184d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:56,238 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:56,238 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=12, startTime=1682024096149; duration=0sec 2023-04-20 20:54:56,238 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:54:56,238 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:54:56,240 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:54:56,240 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:54:56,240 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,240 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5f6177759224493784828f047a93654b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/9281a5a166cc416eb26b8ee47e153b63, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f5593dfd3adc4b59ab07cc8c33418e14, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/70beb237988346d79cf27097c00e1d26] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=44.1 K 2023-04-20 20:54:56,241 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5f6177759224493784828f047a93654b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682024093643 2023-04-20 20:54:56,241 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9281a5a166cc416eb26b8ee47e153b63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024093683 2023-04-20 20:54:56,241 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f5593dfd3adc4b59ab07cc8c33418e14, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1682024094298 2023-04-20 20:54:56,241 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 70beb237988346d79cf27097c00e1d26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682024095429 2023-04-20 20:54:56,249 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#197 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:54:56,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024156254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024156256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,259 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/cf7ea7c91d284a66ae8355f94956ca80 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cf7ea7c91d284a66ae8355f94956ca80 2023-04-20 20:54:56,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024156258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,263 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into cf7ea7c91d284a66ae8355f94956ca80(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:56,263 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:56,263 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=12, startTime=1682024096149; duration=0sec 2023-04-20 20:54:56,264 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:56,335 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:56,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:56,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:56,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:56,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:56,339 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,342 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:56,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:56,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,345 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb2f6c0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:56,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:56,348 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,349 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:56,349 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:56,349 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:56,349 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,349 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:56,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:56,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,350 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:56,352 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:56,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:56,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:56,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:56,353 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:56,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024156359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024156359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024156364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:56,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:56,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:56,373 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:56,373 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:56,473 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:56,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:56,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:56,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:56,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:56,476 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:56,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:56,479 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,479 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:56,479 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:56,479 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:56,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b634a16[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:56,483 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:56,483 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:56,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:56,486 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:56,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:56,488 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:56,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:56,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:56,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:56,489 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:56,490 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,490 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,490 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,491 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:56,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:56,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,494 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:56,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,494 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024156563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024156563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024156566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024156585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024156588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,594 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:56,594 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/a6c088104cde482390ccc3424a8814da as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a6c088104cde482390ccc3424a8814da 2023-04-20 20:54:56,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:56,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:56,598 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:56,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:56,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:56,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,601 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,601 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:56,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:56,608 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into a6c088104cde482390ccc3424a8814da(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:54:56,608 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:56,608 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=12, startTime=1682024096149; duration=0sec 2023-04-20 20:54:56,608 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:54:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:56,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:56,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:56,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:56,611 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,611 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,611 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:56,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721a00c8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:56,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:56,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:56,615 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,615 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:56,616 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/6af48c8daeec416a825b5b3705117d78 2023-04-20 20:54:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:56,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:56,621 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:56,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:56,621 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:56,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:56,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:56,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,633 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:56,633 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:56,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:56,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,733 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:56,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:56,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:56,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:56,737 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:56,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:56,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:56,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,740 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,740 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:56,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:56,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:56,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:56,741 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,741 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:56,741 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:56,741 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:56,743 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16eab74b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:56,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:56,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,746 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:56,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:56,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:56,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:56,750 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:56,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:56,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:56,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:56,750 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:56,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:56,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:56,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:56,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,767 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:56,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,767 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:56,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024156865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,867 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:56,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:56,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:56,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024156868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:56,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024156869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:56,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:56,871 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:56,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:56,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:56,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,873 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:56,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:56,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:56,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:56,875 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,875 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:56,875 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:56,875 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:56,877 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:56,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7257a0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:56,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:56,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:56,887 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,887 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:56,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:56,898 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:56,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:56,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:56,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:56,898 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:56,900 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,900 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,900 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:56,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:56,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:56,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,905 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,905 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,905 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:56,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:56,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:56,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:56,905 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:56,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:56,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:56,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:56,905 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:56,906 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:56,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:56,907 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:56,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:56,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,007 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:57,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:57,010 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:57,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:57,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:57,011 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,011 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,012 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:57,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:57,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:57,014 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:57,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:57,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:57,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:57,021 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,021 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:57,022 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:57,022 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba09f02[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:57,024 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:57,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:57,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:57,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:57,025 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:57,028 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:57,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:57,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:57,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:57,028 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:57,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,030 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/8800571806e344c4ba5b5eaa2209aa0a 2023-04-20 20:54:57,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:57,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:57,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:57,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,040 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,040 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,039 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,039 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:57,040 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:57,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,140 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:57,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:57,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:57,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:57,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:57,143 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:57,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,144 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:57,144 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:57,144 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,144 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,145 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:57,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:57,148 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,148 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:57,148 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:57,148 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:57,150 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@499edefe[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:57,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:57,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,153 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,153 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:57,154 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,154 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:57,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:57,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:57,158 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:57,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:57,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:57,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:57,159 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:57,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,161 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,162 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,168 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:57,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:57,168 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,168 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,168 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:57,168 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:57,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:57,168 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,168 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,268 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:57,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:57,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:57,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:57,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:57,271 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:57,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:57,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:57,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,275 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,275 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,275 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:57,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:57,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:57,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:57,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:57,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:57,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:57,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:57,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:57,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:57,279 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,280 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,280 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:57,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0d7869[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:57,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:57,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,282 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:57,282 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:57,283 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,283 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:57,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:57,284 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:57,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:57,287 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:57,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:57,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:57,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:57,287 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:57,294 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,294 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,294 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:57,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:57,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,299 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:57,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,299 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:57,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024157371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024157372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024157374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,399 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:57,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:57,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:57,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:57,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:57,403 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:57,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:57,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:57,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,404 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,404 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:57,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:57,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:57,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:57,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:57,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:57,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:57,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:57,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:57,408 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,408 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125ae1df[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:57,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:57,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,412 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:57,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:57,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:57,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:57,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,414 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:57,414 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,414 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:57,417 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:57,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:57,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:57,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:57,417 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:57,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:57,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:57,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,421 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:57,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:57,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:57,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:57,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:57,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,425 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:57,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,425 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:57,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:57,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/2d267226bb1e4af1a2b67749dca71c40 2023-04-20 20:54:57,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/6af48c8daeec416a825b5b3705117d78 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6af48c8daeec416a825b5b3705117d78 2023-04-20 20:54:57,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6af48c8daeec416a825b5b3705117d78, entries=200, sequenceid=107, filesize=13.9 K 2023-04-20 20:54:57,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/8800571806e344c4ba5b5eaa2209aa0a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8800571806e344c4ba5b5eaa2209aa0a 2023-04-20 20:54:57,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8800571806e344c4ba5b5eaa2209aa0a, entries=150, sequenceid=107, filesize=11.6 K 2023-04-20 20:54:57,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/2d267226bb1e4af1a2b67749dca71c40 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2d267226bb1e4af1a2b67749dca71c40 2023-04-20 20:54:57,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2d267226bb1e4af1a2b67749dca71c40, entries=150, sequenceid=107, filesize=11.6 K 2023-04-20 20:54:57,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bd7cf796b1328b6fd145cda0a337b2e3 in 1280ms, sequenceid=107, compaction requested=false 2023-04-20 20:54:57,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:57,525 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:57,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:57,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:57,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:57,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:57,529 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:57,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:57,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:57,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:57,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,532 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,532 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:57,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:57,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:57,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:57,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:57,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:57,534 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:57,534 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:57,534 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 20:54:57,534 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:57,534 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:57,534 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:57,535 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:57,535 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:57,535 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:57,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:57,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:57,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024157609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024157609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024157713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024157714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024157916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:57,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024157916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:57,948 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/f373d035e1bc4b4f8217f7be188254d8 2023-04-20 20:54:58,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57986 deadline: 1682024158218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58032 deadline: 1682024158220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,362 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/ac07bbf54d2945e49be14eb020e1975c 2023-04-20 20:54:58,376 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/8d4cb9886c6542d88b196be126bad07f 2023-04-20 20:54:58,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024158377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,381 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/f373d035e1bc4b4f8217f7be188254d8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f373d035e1bc4b4f8217f7be188254d8 2023-04-20 20:54:58,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:58,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024158380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:58,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024158383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,386 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f373d035e1bc4b4f8217f7be188254d8, entries=150, sequenceid=130, filesize=11.6 K 2023-04-20 20:54:58,387 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/ac07bbf54d2945e49be14eb020e1975c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ac07bbf54d2945e49be14eb020e1975c 2023-04-20 20:54:58,391 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ac07bbf54d2945e49be14eb020e1975c, entries=150, sequenceid=130, filesize=11.6 K 2023-04-20 20:54:58,392 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/8d4cb9886c6542d88b196be126bad07f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/8d4cb9886c6542d88b196be126bad07f 2023-04-20 20:54:58,396 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/8d4cb9886c6542d88b196be126bad07f, entries=150, sequenceid=130, filesize=11.6 K 2023-04-20 20:54:58,397 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for bd7cf796b1328b6fd145cda0a337b2e3 in 863ms, sequenceid=130, compaction requested=true 2023-04-20 20:54:58,397 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:58,397 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:58,398 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,399 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:58,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:58,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ded2419[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:58,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:58,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,402 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:58,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:58,403 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:58,404 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:58,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:58,406 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:58,406 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:58,406 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:58,406 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:58,406 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:58,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:58,410 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,410 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:58,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:58,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:58,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:58,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:58,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:58,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:58,415 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,415 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,415 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:58,415 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:58,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:58,515 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:58,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:58,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:58,519 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:58,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:58,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:58,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:58,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,520 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,520 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:58,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:58,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:58,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:58,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:58,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:58,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:58,523 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:58,523 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:58,539 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9c30096c1a9f491f9d0ade9f797915eb 2023-04-20 20:54:58,553 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/d73c2cbc89ee4a728c30dc59e0f99dd3 2023-04-20 20:54:58,566 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/b42de5a2dc914f22b6369c0c488db505 2023-04-20 20:54:58,571 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9c30096c1a9f491f9d0ade9f797915eb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9c30096c1a9f491f9d0ade9f797915eb 2023-04-20 20:54:58,576 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9c30096c1a9f491f9d0ade9f797915eb, entries=150, sequenceid=143, filesize=11.8 K 2023-04-20 20:54:58,577 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/d73c2cbc89ee4a728c30dc59e0f99dd3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d73c2cbc89ee4a728c30dc59e0f99dd3 2023-04-20 20:54:58,581 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d73c2cbc89ee4a728c30dc59e0f99dd3, entries=150, sequenceid=143, filesize=11.8 K 2023-04-20 20:54:58,582 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/b42de5a2dc914f22b6369c0c488db505 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b42de5a2dc914f22b6369c0c488db505 2023-04-20 20:54:58,589 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b42de5a2dc914f22b6369c0c488db505, entries=150, sequenceid=143, filesize=11.8 K 2023-04-20 20:54:58,590 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for bd7cf796b1328b6fd145cda0a337b2e3 in 67ms, sequenceid=143, compaction requested=true 2023-04-20 20:54:58,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:58,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:58,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,599 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:58,599 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,599 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:58,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42bb6c74[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:58,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:58,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:58,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:58,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:58,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:58,609 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:58,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:58,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:58,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:58,609 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:58,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:58,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:58,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:58,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:58,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:58,618 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:58,618 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:58,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:58,718 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:58,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:58,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:58,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:58,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:58,723 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:58,723 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:58,723 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:58,723 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:58,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,725 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:58,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:58,725 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:58,725 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:58,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:58,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:58,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:58,727 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:58,727 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:58,727 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-20 20:54:58,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:58,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:58,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:58,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:58,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:58,729 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:54:58,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:58,733 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:58,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024098755, max timestamp=1682024098765Num uniques 859; 2023-04-20 20:54:58,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:58,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024098755, max timestamp=1682024098761Num uniques 859; 2023-04-20 20:54:58,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024098755, max timestamp=1682024098765Num uniques 859; 2023-04-20 20:54:58,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:58,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:54:58,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024098800, max timestamp=1682024098817Num uniques 1018; 2023-04-20 20:54:58,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:54:58,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024098800, max timestamp=1682024098817Num uniques 1018; 2023-04-20 20:54:58,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:54:58,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024098800, max timestamp=1682024098817Num uniques 1018; 2023-04-20 20:54:58,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 20:54:58,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024098851, max timestamp=1682024098859Num uniques 1077; 2023-04-20 20:54:58,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024098851, max timestamp=1682024098859Num uniques 1077; 2023-04-20 20:54:58,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024098851, max timestamp=1682024098859Num uniques 1027; 2023-04-20 20:54:58,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-20 20:54:58,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:54:58,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-20 20:54:58,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682024098883, max timestamp=1682024098894Num uniques 1036; 2023-04-20 20:54:58,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-20 20:54:58,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682024098883, max timestamp=1682024098894Num uniques 1036; 2023-04-20 20:54:58,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682024098883, max timestamp=1682024098894Num uniques 1036; 2023-04-20 20:54:58,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-20 20:54:58,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-20 20:54:58,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024098934, max timestamp=1682024098943Num uniques 1045; 2023-04-20 20:54:58,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024098934, max timestamp=1682024098943Num uniques 1045; 2023-04-20 20:54:58,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024098934, max timestamp=1682024098943Num uniques 1045; 2023-04-20 20:54:58,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-20 20:54:58,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-20 20:54:58,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-20 20:54:58,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024098975, max timestamp=1682024098985Num uniques 1054; 2023-04-20 20:54:58,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:54:58,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024098975, max timestamp=1682024098985Num uniques 1054; 2023-04-20 20:54:58,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024098975, max timestamp=1682024098985Num uniques 1054; 2023-04-20 20:54:58,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:54:58,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:54:59,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682024099000, max timestamp=1682024099016Num uniques 1013; 2023-04-20 20:54:59,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682024099000, max timestamp=1682024099016Num uniques 1013; 2023-04-20 20:54:59,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682024099000, max timestamp=1682024099016Num uniques 1013; 2023-04-20 20:54:59,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-20 20:54:59,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-20 20:54:59,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-20 20:54:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682024099040, max timestamp=1682024099053Num uniques 1072; 2023-04-20 20:54:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-20 20:54:59,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682024099040, max timestamp=1682024099053Num uniques 1072; 2023-04-20 20:54:59,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-20 20:54:59,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682024099040, max timestamp=1682024099053Num uniques 1072; 2023-04-20 20:54:59,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-20 20:54:59,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024099062, max timestamp=1682024099109Num uniques 1081; 2023-04-20 20:54:59,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-20 20:54:59,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024099062, max timestamp=1682024099109Num uniques 1081; 2023-04-20 20:54:59,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-20 20:54:59,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024099062, max timestamp=1682024099109Num uniques 1081; 2023-04-20 20:54:59,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-20 20:54:59,145 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/528470999c7c454b93076843eaeec0f7 2023-04-20 20:54:59,156 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/281236c2db52483688e2b1506b1926b3 2023-04-20 20:54:59,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682024099156, max timestamp=1682024099169Num uniques 1090; 2023-04-20 20:54:59,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682024099156, max timestamp=1682024099169Num uniques 1090; 2023-04-20 20:54:59,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-20 20:54:59,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-20 20:54:59,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8690, totalHeapSize=20000, min timestamp=1682024099156, max timestamp=1682024099169Num uniques 1090; 2023-04-20 20:54:59,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49910, new segment data size=8690, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-20 20:54:59,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024099182, max timestamp=1682024099204Num uniques 1049; 2023-04-20 20:54:59,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024099182, max timestamp=1682024099204Num uniques 1049; 2023-04-20 20:54:59,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024099182, max timestamp=1682024099204Num uniques 1049; 2023-04-20 20:54:59,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:54:59,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:54:59,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:54:59,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:59,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024159243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682024099237, max timestamp=1682024099246Num uniques 1108; 2023-04-20 20:54:59,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-20 20:54:59,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682024099237, max timestamp=1682024099246Num uniques 1108; 2023-04-20 20:54:59,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-20 20:54:59,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682024099237, max timestamp=1682024099246Num uniques 1108; 2023-04-20 20:54:59,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-20 20:54:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682024099302, max timestamp=1682024099324Num uniques 1017; 2023-04-20 20:54:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682024099302, max timestamp=1682024099324Num uniques 1017; 2023-04-20 20:54:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-20 20:54:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682024099302, max timestamp=1682024099324Num uniques 1017; 2023-04-20 20:54:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-20 20:54:59,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-20 20:54:59,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682024099361, max timestamp=1682024099372Num uniques 1026; 2023-04-20 20:54:59,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682024099361, max timestamp=1682024099372Num uniques 1026; 2023-04-20 20:54:59,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-20 20:54:59,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682024099361, max timestamp=1682024099372Num uniques 1026; 2023-04-20 20:54:59,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-20 20:54:59,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-20 20:54:59,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024099386, max timestamp=1682024099407Num uniques 1035; 2023-04-20 20:54:59,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024099386, max timestamp=1682024099407Num uniques 1035; 2023-04-20 20:54:59,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-20 20:54:59,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-20 20:54:59,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024099386, max timestamp=1682024099407Num uniques 1035; 2023-04-20 20:54:59,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-20 20:54:59,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682024099438, max timestamp=1682024099446Num uniques 1094; 2023-04-20 20:54:59,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-20 20:54:59,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682024099438, max timestamp=1682024099446Num uniques 1094; 2023-04-20 20:54:59,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-20 20:54:59,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682024099438, max timestamp=1682024099446Num uniques 1094; 2023-04-20 20:54:59,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-20 20:54:59,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682024099499, max timestamp=1682024099511Num uniques 1053; 2023-04-20 20:54:59,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-20 20:54:59,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682024099499, max timestamp=1682024099511Num uniques 1053; 2023-04-20 20:54:59,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682024099499, max timestamp=1682024099511Num uniques 1053; 2023-04-20 20:54:59,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-20 20:54:59,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-20 20:54:59,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682024099530, max timestamp=1682024099544Num uniques 1012; 2023-04-20 20:54:59,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682024099530, max timestamp=1682024099544Num uniques 1012; 2023-04-20 20:54:59,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-20 20:54:59,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682024099530, max timestamp=1682024099544Num uniques 1012; 2023-04-20 20:54:59,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-20 20:54:59,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-20 20:54:59,569 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/30abd2469e6e48ddb06ba3d801d46a60 2023-04-20 20:54:59,574 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/528470999c7c454b93076843eaeec0f7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/528470999c7c454b93076843eaeec0f7 2023-04-20 20:54:59,579 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/528470999c7c454b93076843eaeec0f7, entries=150, sequenceid=150, filesize=11.8 K 2023-04-20 20:54:59,580 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/281236c2db52483688e2b1506b1926b3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/281236c2db52483688e2b1506b1926b3 2023-04-20 20:54:59,583 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/281236c2db52483688e2b1506b1926b3, entries=150, sequenceid=150, filesize=11.8 K 2023-04-20 20:54:59,584 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/30abd2469e6e48ddb06ba3d801d46a60 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/30abd2469e6e48ddb06ba3d801d46a60 2023-04-20 20:54:59,588 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/30abd2469e6e48ddb06ba3d801d46a60, entries=150, sequenceid=150, filesize=11.8 K 2023-04-20 20:54:59,603 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=114.05 KB/116790 for bd7cf796b1328b6fd145cda0a337b2e3 in 876ms, sequenceid=150, compaction requested=true 2023-04-20 20:54:59,603 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:54:59,603 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=127.47 KB heapSize=360.09 KB 2023-04-20 20:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:59,603 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:59,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:54:59,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:59,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:54:59,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:59,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:54:59,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:54:59,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,605 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23149a7a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:59,610 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:59,610 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:59,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:59,615 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:59,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:59,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:59,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:59,619 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:59,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:59,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:59,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:59,619 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:59,621 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,621 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,621 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,623 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024159624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024159625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:59,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:59,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,631 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:59,631 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:59,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:59,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:54:59,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/60bd417bd26e487eb3ba33653e6d2a14 2023-04-20 20:54:59,655 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/081d59ddf5804adbbbf2ac5b5754bc2a 2023-04-20 20:54:59,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:59,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024159730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,732 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:59,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:59,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:59,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024159731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:59,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:59,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:59,735 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:59,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:59,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:59,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:59,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,737 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:59,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:59,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:59,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:59,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:59,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:59,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:59,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:59,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:59,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:59,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:59,742 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:59,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6605d3de[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:59,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:59,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,745 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:59,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:59,747 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:59,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:59,749 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:59,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:59,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:59,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:59,750 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:59,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:59,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,752 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,758 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:59,758 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:59,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:59,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:59,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:59,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:59,759 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:59,759 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,759 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:59,759 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:59,759 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,859 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:59,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:59,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:59,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:59,863 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:59,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:59,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:59,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:59,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,865 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,865 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:59,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:59,865 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,865 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:59,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:59,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:59,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:59,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:59,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:59,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:59,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:59,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:54:59,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72dcfeeb[Count = 0] remaining members to acquire global barrier 2023-04-20 20:54:59,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:54:59,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,875 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:54:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:54:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:54:59,875 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,875 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:54:59,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:54:59,877 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:54:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:54:59,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:54:59,880 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:54:59,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:54:59,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:54:59,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:54:59,880 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:54:59,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:59,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:59,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,884 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:54:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,886 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:59,886 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:54:59,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:59,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:54:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:54:59,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:54:59,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:54:59,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,887 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:54:59,887 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:54:59,887 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:54:59,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:59,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024159932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:54:59,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024159934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,987 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:54:59,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:54:59,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:54:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:54:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:54:59,991 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:54:59,991 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:59,991 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:54:59,991 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:54:59,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,994 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:54:59,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:54:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:54:59,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:54:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:54:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:54:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:54:59,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:54:59,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:54:59,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:54:59,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:59,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:54:59,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:54:59,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:54:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:54:59,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,998 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:54:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:54:59,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:54:59,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:54:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:54:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:54:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:54:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:54:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:00,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:00,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b9de408[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:00,000 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:00,000 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:00,001 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:00,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:00,001 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,001 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:00,001 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:00,003 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,003 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:00,003 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:00,003 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:00,006 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:00,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:00,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:00,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:00,006 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:00,008 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,008 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:00,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,009 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:00,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:00,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:00,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:00,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:00,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:00,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:00,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,014 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:00,014 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:00,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:00,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/ec25627a390c40388d591033a1d05d12 2023-04-20 20:55:00,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/60bd417bd26e487eb3ba33653e6d2a14 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/60bd417bd26e487eb3ba33653e6d2a14 2023-04-20 20:55:00,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/60bd417bd26e487eb3ba33653e6d2a14, entries=250, sequenceid=477, filesize=16.7 K 2023-04-20 20:55:00,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/081d59ddf5804adbbbf2ac5b5754bc2a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/081d59ddf5804adbbbf2ac5b5754bc2a 2023-04-20 20:55:00,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/081d59ddf5804adbbbf2ac5b5754bc2a, entries=150, sequenceid=477, filesize=11.9 K 2023-04-20 20:55:00,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/ec25627a390c40388d591033a1d05d12 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ec25627a390c40388d591033a1d05d12 2023-04-20 20:55:00,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ec25627a390c40388d591033a1d05d12, entries=150, sequenceid=477, filesize=11.9 K 2023-04-20 20:55:00,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.05 KB/368688, currentSize=53.67 KB/54960 for bd7cf796b1328b6fd145cda0a337b2e3 in 488ms, sequenceid=477, compaction requested=true 2023-04-20 20:55:00,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:00,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:00,092 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:00,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:00,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 20:55:00,092 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:00,094 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 79274 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:00,094 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:55:00,094 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:00,094 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a6c088104cde482390ccc3424a8814da, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6af48c8daeec416a825b5b3705117d78, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f373d035e1bc4b4f8217f7be188254d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9c30096c1a9f491f9d0ade9f797915eb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/528470999c7c454b93076843eaeec0f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/60bd417bd26e487eb3ba33653e6d2a14] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=77.4 K 2023-04-20 20:55:00,095 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72054 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:00,095 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:55:00,095 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:00,095 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/31738304aa254b0cbdb532b61edf184d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8800571806e344c4ba5b5eaa2209aa0a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ac07bbf54d2945e49be14eb020e1975c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d73c2cbc89ee4a728c30dc59e0f99dd3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/281236c2db52483688e2b1506b1926b3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/081d59ddf5804adbbbf2ac5b5754bc2a] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=70.4 K 2023-04-20 20:55:00,095 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a6c088104cde482390ccc3424a8814da, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682024095429 2023-04-20 20:55:00,096 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 31738304aa254b0cbdb532b61edf184d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682024095429 2023-04-20 20:55:00,096 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6af48c8daeec416a825b5b3705117d78, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682024096078 2023-04-20 20:55:00,096 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8800571806e344c4ba5b5eaa2209aa0a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682024096078 2023-04-20 20:55:00,097 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f373d035e1bc4b4f8217f7be188254d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682024096253 2023-04-20 20:55:00,097 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ac07bbf54d2945e49be14eb020e1975c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682024096253 2023-04-20 20:55:00,097 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d73c2cbc89ee4a728c30dc59e0f99dd3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682024097605 2023-04-20 20:55:00,097 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9c30096c1a9f491f9d0ade9f797915eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682024097605 2023-04-20 20:55:00,098 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 281236c2db52483688e2b1506b1926b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682024098724 2023-04-20 20:55:00,098 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 528470999c7c454b93076843eaeec0f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682024098724 2023-04-20 20:55:00,098 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 081d59ddf5804adbbbf2ac5b5754bc2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682024099570 2023-04-20 20:55:00,098 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 60bd417bd26e487eb3ba33653e6d2a14, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682024099565 2023-04-20 20:55:00,114 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:00,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:00,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:00,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:00,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:00,124 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:00,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:00,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:00,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:00,125 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,126 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:00,128 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#213 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:00,129 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#212 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:00,130 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,129 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:00,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:00,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:00,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:00,132 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,132 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:00,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:00,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:00,140 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:00,140 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:00,140 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:00,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:00,144 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:00,144 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:00,144 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:00,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:00,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:00,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024160277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024100271, max timestamp=1682024100281Num uniques 859; 2023-04-20 20:55:00,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:00,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024100271, max timestamp=1682024100281Num uniques 859; 2023-04-20 20:55:00,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024100271, max timestamp=1682024100281Num uniques 859; 2023-04-20 20:55:00,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:00,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:00,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024160346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024160382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,386 DEBUG [Thread-1441] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:00,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024160383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024160392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,396 DEBUG [Thread-1439] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:00,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024160404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,408 DEBUG [Thread-1445] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:00,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024160451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,568 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/0006b2d1ae9b49a6840dac80b555a67e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/0006b2d1ae9b49a6840dac80b555a67e 2023-04-20 20:55:00,569 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/2750598b893f4374b8ec14340fa50506 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2750598b893f4374b8ec14340fa50506 2023-04-20 20:55:00,570 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9ff0150432ec42d19c0cfdd5abed8f6c 2023-04-20 20:55:00,574 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into 0006b2d1ae9b49a6840dac80b555a67e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:00,574 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:00,574 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=10, startTime=1682024100092; duration=0sec 2023-04-20 20:55:00,574 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:00,575 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:00,577 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into 2750598b893f4374b8ec14340fa50506(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:00,577 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:00,577 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=10, startTime=1682024100092; duration=0sec 2023-04-20 20:55:00,577 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:00,580 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72054 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:00,580 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:55:00,580 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:00,580 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cf7ea7c91d284a66ae8355f94956ca80, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2d267226bb1e4af1a2b67749dca71c40, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/8d4cb9886c6542d88b196be126bad07f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b42de5a2dc914f22b6369c0c488db505, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/30abd2469e6e48ddb06ba3d801d46a60, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ec25627a390c40388d591033a1d05d12] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=70.4 K 2023-04-20 20:55:00,580 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting cf7ea7c91d284a66ae8355f94956ca80, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682024095429 2023-04-20 20:55:00,581 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2d267226bb1e4af1a2b67749dca71c40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682024096078 2023-04-20 20:55:00,581 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8d4cb9886c6542d88b196be126bad07f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682024096253 2023-04-20 20:55:00,582 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b42de5a2dc914f22b6369c0c488db505, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682024097605 2023-04-20 20:55:00,583 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 30abd2469e6e48ddb06ba3d801d46a60, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682024098724 2023-04-20 20:55:00,583 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ec25627a390c40388d591033a1d05d12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682024099570 2023-04-20 20:55:00,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024160589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,605 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#216 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:00,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024160654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024160893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:00,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024160958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:00,995 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/b2d87e76493d456c999d2ff81c83a7c7 2023-04-20 20:55:01,013 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/ddbf094abe364ac694de00ab0698cd7b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ddbf094abe364ac694de00ab0698cd7b 2023-04-20 20:55:01,017 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into ddbf094abe364ac694de00ab0698cd7b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:01,017 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:01,017 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=10, startTime=1682024100092; duration=0sec 2023-04-20 20:55:01,017 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:01,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:01,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024161396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,405 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/590e1c0fcd694799bba9723205335f2e 2023-04-20 20:55:01,409 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9ff0150432ec42d19c0cfdd5abed8f6c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9ff0150432ec42d19c0cfdd5abed8f6c 2023-04-20 20:55:01,413 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9ff0150432ec42d19c0cfdd5abed8f6c, entries=150, sequenceid=488, filesize=11.9 K 2023-04-20 20:55:01,414 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/b2d87e76493d456c999d2ff81c83a7c7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b2d87e76493d456c999d2ff81c83a7c7 2023-04-20 20:55:01,417 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b2d87e76493d456c999d2ff81c83a7c7, entries=150, sequenceid=488, filesize=11.9 K 2023-04-20 20:55:01,418 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/590e1c0fcd694799bba9723205335f2e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/590e1c0fcd694799bba9723205335f2e 2023-04-20 20:55:01,421 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/590e1c0fcd694799bba9723205335f2e, entries=150, sequenceid=488, filesize=11.9 K 2023-04-20 20:55:01,422 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bd7cf796b1328b6fd145cda0a337b2e3 in 1278ms, sequenceid=488, compaction requested=false 2023-04-20 20:55:01,422 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:01,422 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:01,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:01,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:01,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:01,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdd744b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:01,426 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:01,426 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:01,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:01,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:01,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:01,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:01,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:01,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:01,433 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:01,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:01,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:01,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:01,434 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:01,436 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,436 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,436 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:01,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:01,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,442 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:01,442 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:01,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:01,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:01,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 20:55:01,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:01,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:01,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:01,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:01,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:01,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:01,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:01,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024161486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,542 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:01,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:01,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:01,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:01,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:01,546 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:01,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:01,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:01,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,548 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,548 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,548 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:01,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:01,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:01,549 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,550 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:01,550 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:01,550 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:01,551 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,552 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,552 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:01,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@240ec12d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:01,553 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:01,553 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,555 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:01,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:01,559 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:01,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:01,562 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:01,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:01,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:01,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:01,562 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:01,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,564 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:01,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,570 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:01,570 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:01,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024161590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,670 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:01,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:01,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:01,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:01,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:01,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:01,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:01,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:01,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:01,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:01,677 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:01,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:01,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:01,679 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:01,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31b048e1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:01,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:01,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:01,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:01,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:01,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,683 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:01,684 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:01,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:01,687 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:01,687 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:01,687 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:01,687 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:01,687 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:01,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,688 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:01,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:01,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,695 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:01,695 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:01,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,795 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:01,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:01,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:01,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024161794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:01,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:01,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:01,798 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:01,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:01,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:01,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,800 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:01,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:01,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:01,801 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,801 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:01,801 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:01,802 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:01,805 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,805 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,805 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,805 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:01,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3509477b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:01,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:01,807 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,807 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:01,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:01,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:01,811 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:01,811 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:01,811 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:01,811 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:01,811 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:01,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,813 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,817 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,817 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,817 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:01,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:01,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,818 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:01,818 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:01,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,873 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/6adebec56b48432f8b1b83dfaa7bf753 2023-04-20 20:55:01,918 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:01,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:01,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:01,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:01,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:01,922 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:01,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:01,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:01,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:01,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:01,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:01,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,927 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32f102bd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:01,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:01,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:01,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:01,933 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:01,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:01,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:01,936 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:01,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:01,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:01,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:01,937 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:01,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:01,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:01,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,940 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:01,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,945 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:01,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:01,946 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:01,946 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:01,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,046 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,050 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,052 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,053 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,053 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:02,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,055 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,055 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60a4aa67[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,061 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,061 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,064 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,066 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,067 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,074 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,076 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,077 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,079 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,080 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,080 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:02,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:02,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024162098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,180 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,184 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,185 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,187 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,187 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,187 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,187 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,189 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@536011d2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,191 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,191 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,193 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,193 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,193 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,197 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,197 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,199 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,199 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,199 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,200 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,203 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,203 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,203 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,203 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,203 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,203 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,203 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:02,203 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/c454eaaa176d4a908e3e4776a83d35ba 2023-04-20 20:55:02,304 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/1c6ac6c1673240dbb9cf01da52413b47 2023-04-20 20:55:02,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,308 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,323 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/6adebec56b48432f8b1b83dfaa7bf753 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6adebec56b48432f8b1b83dfaa7bf753 2023-04-20 20:55:02,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cadb65a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,328 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,328 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,329 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,330 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,330 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6adebec56b48432f8b1b83dfaa7bf753, entries=200, sequenceid=528, filesize=14.3 K 2023-04-20 20:55:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,335 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,335 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/c454eaaa176d4a908e3e4776a83d35ba as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c454eaaa176d4a908e3e4776a83d35ba 2023-04-20 20:55:02,339 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,339 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,339 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c454eaaa176d4a908e3e4776a83d35ba, entries=150, sequenceid=528, filesize=11.9 K 2023-04-20 20:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/1c6ac6c1673240dbb9cf01da52413b47 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1c6ac6c1673240dbb9cf01da52413b47 2023-04-20 20:55:02,346 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,346 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,346 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,346 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,347 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,347 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,347 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:02,348 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,348 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,348 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:02,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1c6ac6c1673240dbb9cf01da52413b47, entries=150, sequenceid=528, filesize=11.9 K 2023-04-20 20:55:02,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,349 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for bd7cf796b1328b6fd145cda0a337b2e3 in 884ms, sequenceid=528, compaction requested=true 2023-04-20 20:55:02,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:02,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:02,350 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:02,350 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:02,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:02,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:02,355 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:02,356 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:02,356 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:55:02,356 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:55:02,356 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,356 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,356 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2750598b893f4374b8ec14340fa50506, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9ff0150432ec42d19c0cfdd5abed8f6c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6adebec56b48432f8b1b83dfaa7bf753] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=38.4 K 2023-04-20 20:55:02,356 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/0006b2d1ae9b49a6840dac80b555a67e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b2d87e76493d456c999d2ff81c83a7c7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c454eaaa176d4a908e3e4776a83d35ba] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=36.0 K 2023-04-20 20:55:02,357 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2750598b893f4374b8ec14340fa50506, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682024099570 2023-04-20 20:55:02,357 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0006b2d1ae9b49a6840dac80b555a67e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682024099570 2023-04-20 20:55:02,357 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9ff0150432ec42d19c0cfdd5abed8f6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682024099613 2023-04-20 20:55:02,357 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b2d87e76493d456c999d2ff81c83a7c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682024099613 2023-04-20 20:55:02,357 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6adebec56b48432f8b1b83dfaa7bf753, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682024100319 2023-04-20 20:55:02,358 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c454eaaa176d4a908e3e4776a83d35ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682024100319 2023-04-20 20:55:02,364 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#221 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:02,373 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#222 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:02,399 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/6ee534af9a7c43d68ee655da3ee9cce5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6ee534af9a7c43d68ee655da3ee9cce5 2023-04-20 20:55:02,405 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into 6ee534af9a7c43d68ee655da3ee9cce5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:02,405 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:02,405 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=13, startTime=1682024102350; duration=0sec 2023-04-20 20:55:02,405 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:02,405 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:02,406 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:02,407 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:55:02,407 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,407 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ddbf094abe364ac694de00ab0698cd7b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/590e1c0fcd694799bba9723205335f2e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1c6ac6c1673240dbb9cf01da52413b47] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=36.0 K 2023-04-20 20:55:02,407 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ddbf094abe364ac694de00ab0698cd7b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682024099570 2023-04-20 20:55:02,407 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 590e1c0fcd694799bba9723205335f2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682024099613 2023-04-20 20:55:02,408 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 1c6ac6c1673240dbb9cf01da52413b47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682024100319 2023-04-20 20:55:02,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:02,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 20:55:02,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:02,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:02,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:02,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:02,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:02,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:02,416 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:02,438 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/c41f53e77a7a4bcbb63048575f86c5e7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/c41f53e77a7a4bcbb63048575f86c5e7 2023-04-20 20:55:02,442 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into c41f53e77a7a4bcbb63048575f86c5e7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:02,442 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:02,442 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=13, startTime=1682024102350; duration=0sec 2023-04-20 20:55:02,442 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:02,449 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,452 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,453 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,453 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:02,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,455 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,456 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,456 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,456 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f3ad98[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,462 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,462 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024102432, max timestamp=1682024102465Num uniques 859; 2023-04-20 20:55:02,467 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:02,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,467 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024102432, max timestamp=1682024102465Num uniques 859; 2023-04-20 20:55:02,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024102432, max timestamp=1682024102465Num uniques 859; 2023-04-20 20:55:02,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:02,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:02,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,475 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,475 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:02,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,476 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,476 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:02,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024162509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,575 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,579 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,581 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,582 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,582 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,582 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,583 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,583 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,583 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,583 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,583 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,583 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,583 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,584 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5341cab3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,586 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,586 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,587 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,591 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,593 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,593 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,600 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:02,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,600 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:02,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:02,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024162601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:02,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024162612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,700 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,703 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,704 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,704 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,705 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,705 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,705 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,705 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,708 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,708 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,708 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,708 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,711 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fcaf88[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,714 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,715 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,717 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,718 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,719 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,719 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,719 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,720 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,721 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,724 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,724 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,724 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,724 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,725 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,725 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:02,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:02,798 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/ef51c91de2a14c88917e60be6cf33977 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ef51c91de2a14c88917e60be6cf33977 2023-04-20 20:55:02,803 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into ef51c91de2a14c88917e60be6cf33977(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:02,803 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:02,803 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=13, startTime=1682024102350; duration=0sec 2023-04-20 20:55:02,803 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:02,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:02,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024162813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,825 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,828 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/dc2749ba1fa044af85ff74cd0a7e33d8 2023-04-20 20:55:02,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,832 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,832 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,832 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:02,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,836 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,836 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4122d6ab[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,841 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,841 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,843 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,846 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,846 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,846 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,846 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,846 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,849 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:02,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,852 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,852 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:02,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,952 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:02,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:02,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:02,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:02,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:02,956 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:02,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:02,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:02,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,959 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,959 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,959 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,959 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:02,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:02,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:02,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:02,961 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,961 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:02,961 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:02,961 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:02,963 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a363c7a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:02,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:02,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,966 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:02,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:02,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:02,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:02,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:02,970 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:02,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:02,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:02,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:02,970 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:02,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,973 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:02,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:02,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:02,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:02,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:02,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:02,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:02,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:02,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:02,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:02,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:02,978 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:02,977 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:02,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:02,978 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:03,078 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:03,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:03,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:03,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:03,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:03,081 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:03,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:03,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:03,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,083 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:03,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:03,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:03,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:03,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:03,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:03,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:03,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:03,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:03,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:03,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:03,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177f73dd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:03,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:03,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,091 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,091 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:03,091 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:03,091 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,091 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:03,091 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:03,093 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,093 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:03,093 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,093 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:03,095 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:03,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:03,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:03,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:03,095 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:03,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:03,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,098 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:03,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:03,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:03,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,101 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:03,101 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:03,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:03,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024163116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,201 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:03,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:03,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:03,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:03,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:03,205 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:03,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:03,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:03,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,208 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:03,208 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:03,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:03,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:03,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:03,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:03,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:03,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:03,210 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:03,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:03,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,213 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:03,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721a1e55[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:03,214 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:03,214 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,217 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:03,217 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,217 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:03,220 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:03,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:03,222 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:03,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:03,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:03,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:03,222 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:03,224 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,224 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,224 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,227 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,227 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:03,227 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:03,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:03,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:03,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:03,228 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:03,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:03,228 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:03,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:03,228 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:03,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/68f403d86fd84daea4a942d4ffca2d9d 2023-04-20 20:55:03,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/e09a39bc480e41aa8898bb45ebe5c2af 2023-04-20 20:55:03,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/dc2749ba1fa044af85ff74cd0a7e33d8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/dc2749ba1fa044af85ff74cd0a7e33d8 2023-04-20 20:55:03,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/dc2749ba1fa044af85ff74cd0a7e33d8, entries=200, sequenceid=540, filesize=14.3 K 2023-04-20 20:55:03,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/68f403d86fd84daea4a942d4ffca2d9d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/68f403d86fd84daea4a942d4ffca2d9d 2023-04-20 20:55:03,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/68f403d86fd84daea4a942d4ffca2d9d, entries=150, sequenceid=540, filesize=11.9 K 2023-04-20 20:55:03,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/e09a39bc480e41aa8898bb45ebe5c2af as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e09a39bc480e41aa8898bb45ebe5c2af 2023-04-20 20:55:03,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e09a39bc480e41aa8898bb45ebe5c2af, entries=150, sequenceid=540, filesize=11.9 K 2023-04-20 20:55:03,279 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bd7cf796b1328b6fd145cda0a337b2e3 in 870ms, sequenceid=540, compaction requested=false 2023-04-20 20:55:03,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:03,328 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:03,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:03,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:03,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:03,332 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:03,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:03,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:03,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:03,334 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:03,334 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,334 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:03,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:03,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:03,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:03,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:03,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:03,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:03,336 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:03,336 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:03,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:03,605 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:03,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:03,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:03,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024163623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024163623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:03,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024163727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:03,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024163727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,747 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/f16dac3176ef45e6bc035ebeb959ba24 2023-04-20 20:55:03,758 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/41bd778891bd45cb86602fe286f59bd0 2023-04-20 20:55:03,770 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f85d60d9663649f7bdc6d3354ee36846 2023-04-20 20:55:03,774 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/f16dac3176ef45e6bc035ebeb959ba24 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f16dac3176ef45e6bc035ebeb959ba24 2023-04-20 20:55:03,777 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f16dac3176ef45e6bc035ebeb959ba24, entries=150, sequenceid=578, filesize=11.9 K 2023-04-20 20:55:03,778 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/41bd778891bd45cb86602fe286f59bd0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/41bd778891bd45cb86602fe286f59bd0 2023-04-20 20:55:03,782 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/41bd778891bd45cb86602fe286f59bd0, entries=150, sequenceid=578, filesize=11.9 K 2023-04-20 20:55:03,782 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f85d60d9663649f7bdc6d3354ee36846 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f85d60d9663649f7bdc6d3354ee36846 2023-04-20 20:55:03,787 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f85d60d9663649f7bdc6d3354ee36846, entries=150, sequenceid=578, filesize=11.9 K 2023-04-20 20:55:03,787 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for bd7cf796b1328b6fd145cda0a337b2e3 in 451ms, sequenceid=578, compaction requested=true 2023-04-20 20:55:03,788 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:03,788 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:03,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:03,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@255a1933[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:03,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:03,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,793 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:03,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:03,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:03,798 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:03,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:03,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:03,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:03,798 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:03,799 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:03,800 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,800 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:03,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:03,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:03,802 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:03,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:03,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:03,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:03,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:03,807 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:03,807 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:03,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:03,907 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:03,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:03,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:03,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:03,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:03,911 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:03,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:03,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:03,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:03,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:03,913 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,913 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:03,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:03,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:03,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:03,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:03,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:03,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:03,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:03,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:03,914 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:03,914 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:03,914 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:55:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:03,923 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/8c94d54c60ab4ecda8a73e53bb25a05e 2023-04-20 20:55:03,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:03,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:03,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:03,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024163953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:03,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:03,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024163955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024164056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024164058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024164259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024164260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,334 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/d1748f2b516945b79566079b3594f445 2023-04-20 20:55:04,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58072 deadline: 1682024164403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,406 DEBUG [Thread-1439] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:04,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58062 deadline: 1682024164417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,419 DEBUG [Thread-1441] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:04,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58046 deadline: 1682024164448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,450 DEBUG [Thread-1445] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:04,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024164563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:04,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024164564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,746 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/56fff4fd551240bd8ac2605ed85577e0 2023-04-20 20:55:04,752 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/8c94d54c60ab4ecda8a73e53bb25a05e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8c94d54c60ab4ecda8a73e53bb25a05e 2023-04-20 20:55:04,755 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8c94d54c60ab4ecda8a73e53bb25a05e, entries=150, sequenceid=590, filesize=11.9 K 2023-04-20 20:55:04,756 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/d1748f2b516945b79566079b3594f445 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d1748f2b516945b79566079b3594f445 2023-04-20 20:55:04,760 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d1748f2b516945b79566079b3594f445, entries=150, sequenceid=590, filesize=11.9 K 2023-04-20 20:55:04,761 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/56fff4fd551240bd8ac2605ed85577e0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/56fff4fd551240bd8ac2605ed85577e0 2023-04-20 20:55:04,765 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/56fff4fd551240bd8ac2605ed85577e0, entries=150, sequenceid=590, filesize=11.9 K 2023-04-20 20:55:04,765 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bd7cf796b1328b6fd145cda0a337b2e3 in 851ms, sequenceid=590, compaction requested=true 2023-04-20 20:55:04,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:04,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:04,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:04,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:04,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:04,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:04,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:04,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,770 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:04,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e74240e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:04,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:04,772 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,773 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:04,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:04,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:04,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:04,778 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:04,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:04,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:04,781 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:04,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:04,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:04,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:04,781 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:04,782 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,782 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:04,783 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,783 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:04,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:04,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:04,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:04,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:04,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:04,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:04,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:04,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:04,787 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:04,787 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:04,787 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,787 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:04,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:04,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:04,887 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:04,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:04,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:04,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:04,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:04,891 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:04,891 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:04,891 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:04,891 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:04,891 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,891 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:04,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:04,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:04,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:04,893 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:04,893 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:04,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:04,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:04,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:04,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:04,894 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:04,894 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:04,894 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 20:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:05,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:05,070 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:05,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024165082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024165085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024165186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024165189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,307 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/a00e767eb3fc4d9ebc93d43544080e5a 2023-04-20 20:55:05,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024165389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024165392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024165690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:05,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024165695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:05,717 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/677ddd2d3bec4f268b2227b7d3fd9cdc 2023-04-20 20:55:06,160 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/1960ffc4395e40f492f90daa717279e6 2023-04-20 20:55:06,165 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/a00e767eb3fc4d9ebc93d43544080e5a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a00e767eb3fc4d9ebc93d43544080e5a 2023-04-20 20:55:06,173 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a00e767eb3fc4d9ebc93d43544080e5a, entries=150, sequenceid=610, filesize=11.9 K 2023-04-20 20:55:06,174 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/677ddd2d3bec4f268b2227b7d3fd9cdc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/677ddd2d3bec4f268b2227b7d3fd9cdc 2023-04-20 20:55:06,178 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/677ddd2d3bec4f268b2227b7d3fd9cdc, entries=150, sequenceid=610, filesize=11.9 K 2023-04-20 20:55:06,179 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/1960ffc4395e40f492f90daa717279e6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1960ffc4395e40f492f90daa717279e6 2023-04-20 20:55:06,183 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1960ffc4395e40f492f90daa717279e6, entries=150, sequenceid=610, filesize=11.9 K 2023-04-20 20:55:06,184 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bd7cf796b1328b6fd145cda0a337b2e3 in 1290ms, sequenceid=610, compaction requested=true 2023-04-20 20:55:06,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:06,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:06,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:06,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32794586[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:06,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:06,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,189 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:06,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:06,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:06,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:06,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 20:55:06,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:06,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:06,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:06,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:06,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:06,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:06,198 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:06,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:06,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:06,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:06,198 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:06,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/34256c8cef144bdb8c24cf291de7b382 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,207 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:06,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:06,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:06,207 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:06,207 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024166235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024166238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,307 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:06,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:06,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:06,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:06,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:06,311 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:06,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:06,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:06,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:06,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:06,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:06,316 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,316 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:06,316 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:06,316 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:06,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,319 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:06,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b31d2a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:06,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,321 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:06,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:06,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:06,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:06,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:06,323 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:06,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:06,325 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:06,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:06,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:06,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:06,325 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:06,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,328 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,331 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,332 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,332 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,332 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,332 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,332 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:06,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:06,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:06,332 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:06,332 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:06,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024166341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024166344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,432 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:06,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:06,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:06,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:06,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:06,435 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:06,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:06,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:06,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,437 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,438 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,438 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:06,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:06,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:06,439 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,439 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:06,439 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:06,439 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,439 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:06,440 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:06,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d50d8e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:06,443 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:06,443 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,444 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:06,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:06,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:06,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:06,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:06,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,447 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,447 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:06,449 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:06,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:06,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:06,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:06,449 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:06,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,453 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,455 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:06,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,456 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:06,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:06,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,456 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:06,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024166545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024166547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,556 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:06,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:06,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:06,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:06,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:06,559 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:06,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:06,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:06,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,560 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:06,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:06,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:06,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:06,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:06,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:06,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:06,565 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:06,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1afe116e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:06,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,567 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,568 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:06,568 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:06,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:06,568 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:06,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,569 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:06,569 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,569 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:06,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:06,571 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:06,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:06,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:06,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:06,571 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:06,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,573 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,574 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,583 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,583 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,583 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,583 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:06,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:06,583 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,583 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,584 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,584 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,584 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:06,584 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:06,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:06,584 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,584 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/cbe21ef228584fcab15f9bddf7d19d5f 2023-04-20 20:55:06,684 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:06,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:06,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:06,688 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:06,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:06,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:06,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:06,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:06,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:06,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:06,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:06,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:06,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:06,691 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,691 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:06,692 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:06,692 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:06,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:06,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625322c2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:06,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:06,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:06,698 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,698 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:06,700 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,700 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,700 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,700 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:06,702 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:06,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:06,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:06,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:06,702 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:06,704 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,704 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,704 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,704 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,708 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:06,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:06,709 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,709 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:06,709 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,709 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:06,709 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:06,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,809 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:06,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:06,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:06,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:06,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:06,813 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:06,813 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,813 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:06,813 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:06,813 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,813 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,814 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,815 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:06,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:06,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:06,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:06,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:06,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:06,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:06,816 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:06,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:06,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:06,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:06,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8139ef[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:06,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:06,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:06,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:06,823 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:06,823 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,823 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,823 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:06,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:06,826 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:06,826 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:06,826 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:06,826 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:06,826 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:06,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,828 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,829 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:06,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:06,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,832 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,832 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,833 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:06,833 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:06,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024166852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:06,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024166852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,933 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:06,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:06,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:06,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:06,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:06,936 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:06,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:06,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:06,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:06,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:06,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:06,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:06,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:06,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:06,940 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,940 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:06,940 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:06,940 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:06,941 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:06,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2775e04c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:06,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:06,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:06,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:06,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:06,945 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,945 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:06,947 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,948 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:06,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:06,950 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:06,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:06,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:06,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:06,950 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:06,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:06,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:06,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:06,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:06,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:06,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:06,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:06,960 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:06,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:06,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:06,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:06,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:06,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:06,961 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:06,961 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:06,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:06,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:07,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/de105b2c90c545858a01391bd682e648 2023-04-20 20:55:07,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/34256c8cef144bdb8c24cf291de7b382 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/34256c8cef144bdb8c24cf291de7b382 2023-04-20 20:55:07,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/34256c8cef144bdb8c24cf291de7b382, entries=150, sequenceid=623, filesize=11.9 K 2023-04-20 20:55:07,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/cbe21ef228584fcab15f9bddf7d19d5f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/cbe21ef228584fcab15f9bddf7d19d5f 2023-04-20 20:55:07,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/cbe21ef228584fcab15f9bddf7d19d5f, entries=150, sequenceid=623, filesize=11.9 K 2023-04-20 20:55:07,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/de105b2c90c545858a01391bd682e648 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/de105b2c90c545858a01391bd682e648 2023-04-20 20:55:07,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/de105b2c90c545858a01391bd682e648, entries=150, sequenceid=623, filesize=11.9 K 2023-04-20 20:55:07,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for bd7cf796b1328b6fd145cda0a337b2e3 in 855ms, sequenceid=623, compaction requested=true 2023-04-20 20:55:07,051 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:07,051 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:07,051 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:07,051 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:07,051 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:07,051 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:07,053 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76000 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:07,053 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73560 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:07,053 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:55:07,053 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:55:07,053 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,053 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,053 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6ee534af9a7c43d68ee655da3ee9cce5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/dc2749ba1fa044af85ff74cd0a7e33d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f16dac3176ef45e6bc035ebeb959ba24, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8c94d54c60ab4ecda8a73e53bb25a05e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a00e767eb3fc4d9ebc93d43544080e5a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/34256c8cef144bdb8c24cf291de7b382] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=74.2 K 2023-04-20 20:55:07,053 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ef51c91de2a14c88917e60be6cf33977, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/68f403d86fd84daea4a942d4ffca2d9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/41bd778891bd45cb86602fe286f59bd0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d1748f2b516945b79566079b3594f445, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/677ddd2d3bec4f268b2227b7d3fd9cdc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/cbe21ef228584fcab15f9bddf7d19d5f] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=71.8 K 2023-04-20 20:55:07,053 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6ee534af9a7c43d68ee655da3ee9cce5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682024100319 2023-04-20 20:55:07,054 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ef51c91de2a14c88917e60be6cf33977, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682024100319 2023-04-20 20:55:07,054 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting dc2749ba1fa044af85ff74cd0a7e33d8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1682024101474 2023-04-20 20:55:07,054 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 68f403d86fd84daea4a942d4ffca2d9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1682024101483 2023-04-20 20:55:07,054 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f16dac3176ef45e6bc035ebeb959ba24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1682024102502 2023-04-20 20:55:07,054 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 41bd778891bd45cb86602fe286f59bd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1682024102502 2023-04-20 20:55:07,055 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8c94d54c60ab4ecda8a73e53bb25a05e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682024103607 2023-04-20 20:55:07,055 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d1748f2b516945b79566079b3594f445, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682024103607 2023-04-20 20:55:07,055 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a00e767eb3fc4d9ebc93d43544080e5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682024103949 2023-04-20 20:55:07,055 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 677ddd2d3bec4f268b2227b7d3fd9cdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682024103949 2023-04-20 20:55:07,055 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 34256c8cef144bdb8c24cf291de7b382, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682024105081 2023-04-20 20:55:07,055 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting cbe21ef228584fcab15f9bddf7d19d5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682024105081 2023-04-20 20:55:07,061 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:07,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:07,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:07,065 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:07,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:07,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:07,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,066 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#239 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:07,066 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#240 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:07,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:07,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:07,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:07,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:07,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:07,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:07,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:07,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:07,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:07,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:07,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:07,075 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,075 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:07,075 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 20:55:07,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:07,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:07,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:07,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,084 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/86bfa37b4bbf494c9023358165bd4683 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/86bfa37b4bbf494c9023358165bd4683 2023-04-20 20:55:07,088 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into 86bfa37b4bbf494c9023358165bd4683(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:07,088 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:07,088 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=10, startTime=1682024107051; duration=0sec 2023-04-20 20:55:07,088 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:07,088 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:07,090 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73560 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:07,090 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:55:07,090 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,090 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/c41f53e77a7a4bcbb63048575f86c5e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e09a39bc480e41aa8898bb45ebe5c2af, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f85d60d9663649f7bdc6d3354ee36846, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/56fff4fd551240bd8ac2605ed85577e0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1960ffc4395e40f492f90daa717279e6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/de105b2c90c545858a01391bd682e648] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=71.8 K 2023-04-20 20:55:07,090 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c41f53e77a7a4bcbb63048575f86c5e7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682024100319 2023-04-20 20:55:07,091 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e09a39bc480e41aa8898bb45ebe5c2af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1682024101483 2023-04-20 20:55:07,091 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f85d60d9663649f7bdc6d3354ee36846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1682024102502 2023-04-20 20:55:07,092 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 56fff4fd551240bd8ac2605ed85577e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682024103607 2023-04-20 20:55:07,092 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1960ffc4395e40f492f90daa717279e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682024103949 2023-04-20 20:55:07,092 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting de105b2c90c545858a01391bd682e648, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682024105081 2023-04-20 20:55:07,094 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/c1a6dbca15064fbe8998fa36bbed3f47 2023-04-20 20:55:07,104 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:07,110 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/570141df101a4b5e85725e7328f089b8 2023-04-20 20:55:07,117 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/e96321c7bc9b44f4b7b8a5f3e25d7aad as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e96321c7bc9b44f4b7b8a5f3e25d7aad 2023-04-20 20:55:07,124 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into e96321c7bc9b44f4b7b8a5f3e25d7aad(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:07,124 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:07,124 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=10, startTime=1682024107051; duration=0sec 2023-04-20 20:55:07,124 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:07,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:07,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:07,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024167396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024167396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,494 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/e432427e33bb4859bff343b7de6b57fb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/e432427e33bb4859bff343b7de6b57fb 2023-04-20 20:55:07,498 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into e432427e33bb4859bff343b7de6b57fb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:07,498 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:07,498 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=10, startTime=1682024107051; duration=0sec 2023-04-20 20:55:07,498 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:07,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024167500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024167500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,533 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/b6ea27dfb44d4db098226895f5b25013 2023-04-20 20:55:07,537 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/c1a6dbca15064fbe8998fa36bbed3f47 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c1a6dbca15064fbe8998fa36bbed3f47 2023-04-20 20:55:07,543 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c1a6dbca15064fbe8998fa36bbed3f47, entries=150, sequenceid=642, filesize=11.9 K 2023-04-20 20:55:07,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/570141df101a4b5e85725e7328f089b8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/570141df101a4b5e85725e7328f089b8 2023-04-20 20:55:07,548 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/570141df101a4b5e85725e7328f089b8, entries=150, sequenceid=642, filesize=11.9 K 2023-04-20 20:55:07,549 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/b6ea27dfb44d4db098226895f5b25013 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b6ea27dfb44d4db098226895f5b25013 2023-04-20 20:55:07,553 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b6ea27dfb44d4db098226895f5b25013, entries=150, sequenceid=642, filesize=11.9 K 2023-04-20 20:55:07,554 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for bd7cf796b1328b6fd145cda0a337b2e3 in 479ms, sequenceid=642, compaction requested=false 2023-04-20 20:55:07,554 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:07,554 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:07,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,555 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d6ae0ad[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:07,557 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:07,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,560 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:07,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:07,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:07,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:07,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:07,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:07,562 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:07,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:07,565 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:07,565 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:07,565 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:07,565 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:07,565 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:07,567 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,567 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,567 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:07,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,572 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:07,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:07,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:07,572 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:07,572 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:07,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:07,672 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:07,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:07,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:07,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:07,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:07,676 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:07,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:07,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:07,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,679 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:07,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:07,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:07,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:07,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:07,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:07,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:07,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:07,681 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,691 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9d648b24d4084a06bea0e6317a2df088 2023-04-20 20:55:07,701 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/48e91716a66142dc90ae308f054fa447 2023-04-20 20:55:07,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:07,704 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:07,712 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/2ef75740de31409fbf4aacbe6ba80ce4 2023-04-20 20:55:07,718 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9d648b24d4084a06bea0e6317a2df088 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9d648b24d4084a06bea0e6317a2df088 2023-04-20 20:55:07,721 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9d648b24d4084a06bea0e6317a2df088, entries=150, sequenceid=659, filesize=11.9 K 2023-04-20 20:55:07,722 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/48e91716a66142dc90ae308f054fa447 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/48e91716a66142dc90ae308f054fa447 2023-04-20 20:55:07,727 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/48e91716a66142dc90ae308f054fa447, entries=150, sequenceid=659, filesize=11.9 K 2023-04-20 20:55:07,728 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/2ef75740de31409fbf4aacbe6ba80ce4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2ef75740de31409fbf4aacbe6ba80ce4 2023-04-20 20:55:07,731 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2ef75740de31409fbf4aacbe6ba80ce4, entries=150, sequenceid=659, filesize=11.9 K 2023-04-20 20:55:07,732 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=67.09 KB/68700 for bd7cf796b1328b6fd145cda0a337b2e3 in 51ms, sequenceid=659, compaction requested=true 2023-04-20 20:55:07,732 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:07,732 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:07,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:07,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:07,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:07,737 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:07,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:55:07,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:07,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:07,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:07,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:07,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:07,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d48ef0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:07,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:07,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,742 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:07,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:07,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:07,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:07,745 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:07,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:07,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:07,749 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:07,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:07,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:07,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:07,749 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:07,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:07,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:07,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:07,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:07,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:07,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:07,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:07,756 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:07,757 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:07,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:07,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:07,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:07,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:07,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/8f5dad54bcf84376a27b293b11c7fbef 2023-04-20 20:55:07,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024167768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024167772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,857 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:07,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:07,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:07,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:07,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:07,860 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:07,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:07,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:07,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:07,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:07,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:07,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:07,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:07,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:07,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:07,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:07,866 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:07,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55798784[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:07,868 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:07,868 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,870 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:07,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:07,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:07,870 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:07,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024167869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:07,872 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:07,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:07,874 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:07,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:07,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:07,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:07,875 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:07,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:07,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024167874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,876 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,876 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:07,876 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:07,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:07,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:07,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:07,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:07,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:07,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:07,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:07,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:07,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:07,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:07,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:07,881 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:07,881 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:07,981 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:07,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:07,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:07,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:07,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:07,984 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:07,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:07,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:07,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:07,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:07,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:07,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:07,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:07,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:07,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:07,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:07,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:07,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:07,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:07,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:08,000 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d066ae7[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:08,001 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:08,001 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,002 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:08,002 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:08,002 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:08,002 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,003 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:08,004 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:08,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:08,006 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:08,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:08,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:08,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:08,006 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:08,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:08,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:08,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,016 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:08,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,016 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:08,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024168072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:08,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024168077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,116 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:08,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:08,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:08,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:08,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:08,120 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:08,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:08,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:08,121 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,121 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,122 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,122 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,122 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,122 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:08,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:08,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:08,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:08,123 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:08,123 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,123 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:08,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:08,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:08,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:08,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:08,124 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:08,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:08,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:08,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:08,126 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,126 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,126 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,126 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:08,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f39662[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:08,129 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:08,129 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,131 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:08,131 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:08,131 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:08,131 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,131 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:08,132 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,132 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:08,132 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,132 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:08,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:08,135 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:08,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:08,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:08,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:08,135 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:08,137 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,137 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,137 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,137 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,138 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:08,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,141 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:08,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:08,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,141 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:08,142 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:08,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/99707c458550467792f36c9074d451d9 2023-04-20 20:55:08,242 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:08,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:08,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:08,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:08,245 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:08,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:08,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:08,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,248 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,248 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,248 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:08,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:08,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:08,250 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,250 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:08,250 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:08,250 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:08,250 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:08,252 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:08,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c23362[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:08,254 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:08,254 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:08,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:08,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:08,255 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,255 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:08,256 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,256 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:08,256 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,256 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:08,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:08,259 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:08,259 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:08,259 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:08,259 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:08,259 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:08,261 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,261 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,261 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:08,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,265 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:08,265 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:08,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:08,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,366 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:08,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:08,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:08,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:08,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:08,369 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:08,369 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,369 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:08,369 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:08,370 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,370 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,371 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,371 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,371 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,371 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:08,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:08,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:08,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:08,372 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:08,372 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,372 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:08,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:08,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:08,373 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,373 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:08,373 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:08,373 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:08,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:08,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:08,373 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:08,374 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,375 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,375 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:08,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024168376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,376 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c9ecbf9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:08,376 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,378 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,379 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:08,379 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:08,379 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:08,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:08,380 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:08,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:08,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:08,383 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:08,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:08,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:08,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:08,383 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:08,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:08,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024168382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,384 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,384 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,384 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,385 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,388 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,388 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,388 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,388 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:08,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:08,388 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,388 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,389 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,389 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,389 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:08,389 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,389 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:08,389 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,489 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:08,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:08,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:08,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:08,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:08,492 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:08,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:08,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:08,493 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,493 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,497 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:08,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:08,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:08,499 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,499 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:08,499 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:08,499 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:08,501 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,501 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,501 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,501 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:08,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2221d1df[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:08,503 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:08,503 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:08,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:08,506 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:08,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:08,509 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:08,509 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:08,509 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:08,509 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:08,509 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:08,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:08,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,514 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:08,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:08,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,517 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:08,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:08,517 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:08,517 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:08,516 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:08,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:08,517 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:08,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/66f57cca4c3c40abb577dc2b1f4a81e3 2023-04-20 20:55:08,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/8f5dad54bcf84376a27b293b11c7fbef as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8f5dad54bcf84376a27b293b11c7fbef 2023-04-20 20:55:08,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8f5dad54bcf84376a27b293b11c7fbef, entries=250, sequenceid=674, filesize=16.7 K 2023-04-20 20:55:08,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/99707c458550467792f36c9074d451d9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/99707c458550467792f36c9074d451d9 2023-04-20 20:55:08,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/99707c458550467792f36c9074d451d9, entries=150, sequenceid=674, filesize=11.9 K 2023-04-20 20:55:08,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/66f57cca4c3c40abb577dc2b1f4a81e3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/66f57cca4c3c40abb577dc2b1f4a81e3 2023-04-20 20:55:08,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/66f57cca4c3c40abb577dc2b1f4a81e3, entries=150, sequenceid=674, filesize=11.9 K 2023-04-20 20:55:08,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for bd7cf796b1328b6fd145cda0a337b2e3 in 876ms, sequenceid=674, compaction requested=true 2023-04-20 20:55:08,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:08,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:08,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:08,615 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:08,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:08,615 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:08,616 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54272 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:08,616 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:55:08,616 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:08,616 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,616 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:55:08,616 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/e432427e33bb4859bff343b7de6b57fb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c1a6dbca15064fbe8998fa36bbed3f47, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9d648b24d4084a06bea0e6317a2df088, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8f5dad54bcf84376a27b293b11c7fbef] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=53 K 2023-04-20 20:55:08,616 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,616 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/86bfa37b4bbf494c9023358165bd4683, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/570141df101a4b5e85725e7328f089b8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/48e91716a66142dc90ae308f054fa447, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/99707c458550467792f36c9074d451d9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=48.2 K 2023-04-20 20:55:08,617 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e432427e33bb4859bff343b7de6b57fb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682024105081 2023-04-20 20:55:08,617 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 86bfa37b4bbf494c9023358165bd4683, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682024105081 2023-04-20 20:55:08,617 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:08,617 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting c1a6dbca15064fbe8998fa36bbed3f47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1682024106231 2023-04-20 20:55:08,617 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 570141df101a4b5e85725e7328f089b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1682024106231 2023-04-20 20:55:08,617 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9d648b24d4084a06bea0e6317a2df088, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682024107357 2023-04-20 20:55:08,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:08,618 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 48e91716a66142dc90ae308f054fa447, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682024107357 2023-04-20 20:55:08,618 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8f5dad54bcf84376a27b293b11c7fbef, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1682024107711 2023-04-20 20:55:08,618 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 99707c458550467792f36c9074d451d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1682024107711 2023-04-20 20:55:08,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:08,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:08,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:08,621 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:08,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:08,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:08,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:08,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,623 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,623 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:08,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:08,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:08,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:08,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:08,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:08,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:08,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:08,624 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:08,624 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:08,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:08,625 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:55:08,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:08,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:08,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:08,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:08,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:08,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:08,627 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#251 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:08,628 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#252 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:08,644 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9e4a161ae494488aa5e0964c450e917e 2023-04-20 20:55:08,657 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/583cc231d01845b2b5df92502fd8d76c 2023-04-20 20:55:08,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:08,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:08,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:08,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024168898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:08,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:08,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024168899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024169002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024169002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,037 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/c9222f5972de4819b6a936e859864504 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c9222f5972de4819b6a936e859864504 2023-04-20 20:55:09,041 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/274e7805a32a48f9925e53098d61993f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/274e7805a32a48f9925e53098d61993f 2023-04-20 20:55:09,042 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into c9222f5972de4819b6a936e859864504(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:09,042 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:09,042 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=12, startTime=1682024108615; duration=0sec 2023-04-20 20:55:09,043 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:09,043 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:09,044 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:09,044 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:55:09,044 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,044 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e96321c7bc9b44f4b7b8a5f3e25d7aad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b6ea27dfb44d4db098226895f5b25013, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2ef75740de31409fbf4aacbe6ba80ce4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/66f57cca4c3c40abb577dc2b1f4a81e3] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=48.2 K 2023-04-20 20:55:09,045 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e96321c7bc9b44f4b7b8a5f3e25d7aad, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682024105081 2023-04-20 20:55:09,045 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b6ea27dfb44d4db098226895f5b25013, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1682024106231 2023-04-20 20:55:09,046 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2ef75740de31409fbf4aacbe6ba80ce4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682024107357 2023-04-20 20:55:09,046 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 66f57cca4c3c40abb577dc2b1f4a81e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1682024107711 2023-04-20 20:55:09,047 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into 274e7805a32a48f9925e53098d61993f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:09,047 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:09,047 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=12, startTime=1682024108615; duration=0sec 2023-04-20 20:55:09,047 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:09,053 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#256 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:09,070 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/82e38fc393ea437a8cbdbe2f72369bdf 2023-04-20 20:55:09,075 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/9e4a161ae494488aa5e0964c450e917e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9e4a161ae494488aa5e0964c450e917e 2023-04-20 20:55:09,079 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9e4a161ae494488aa5e0964c450e917e, entries=150, sequenceid=691, filesize=11.9 K 2023-04-20 20:55:09,079 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/583cc231d01845b2b5df92502fd8d76c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/583cc231d01845b2b5df92502fd8d76c 2023-04-20 20:55:09,083 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/583cc231d01845b2b5df92502fd8d76c, entries=150, sequenceid=691, filesize=11.9 K 2023-04-20 20:55:09,084 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/82e38fc393ea437a8cbdbe2f72369bdf as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/82e38fc393ea437a8cbdbe2f72369bdf 2023-04-20 20:55:09,089 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/82e38fc393ea437a8cbdbe2f72369bdf, entries=150, sequenceid=691, filesize=11.9 K 2023-04-20 20:55:09,090 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for bd7cf796b1328b6fd145cda0a337b2e3 in 465ms, sequenceid=691, compaction requested=false 2023-04-20 20:55:09,090 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:09,090 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,091 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,091 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,091 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,091 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,092 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ba137f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,096 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,100 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,101 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,102 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,102 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,102 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,107 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,107 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,108 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:09,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 20:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:09,208 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:09,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:09,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:09,213 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:09,213 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,213 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:09,213 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:09,213 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,213 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,216 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,216 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:09,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:09,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:09,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:09,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:09,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:09,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:09,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:09,217 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,218 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:09,218 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:09,218 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,219 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,220 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d277609[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,227 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,227 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024169228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024169229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,234 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,234 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,235 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,236 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,236 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,236 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,240 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,240 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:09,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024169332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024169333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,340 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:09,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:09,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:09,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:09,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:09,343 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:09,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:09,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:09,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,346 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:09,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:09,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:09,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:09,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:09,348 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,350 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dcb357[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,353 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,354 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,357 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,357 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,365 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,365 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,365 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,365 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,367 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,369 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,369 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,369 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,370 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,462 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/15491159496842a0a422a86428efa0b2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/15491159496842a0a422a86428efa0b2 2023-04-20 20:55:09,466 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into 15491159496842a0a422a86428efa0b2(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:09,466 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:09,466 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=12, startTime=1682024108615; duration=0sec 2023-04-20 20:55:09,466 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:09,469 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:09,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:09,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:09,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:09,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:09,473 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:09,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:09,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:09,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:09,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:09,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:09,476 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,476 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:09,476 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:09,476 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,478 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b5f355a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,483 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,484 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,486 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,486 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,491 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,492 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,492 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024169534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024169536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,592 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:09,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:09,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:09,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:09,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:09,596 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:09,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:09,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:09,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,598 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:09,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:09,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:09,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:09,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:09,600 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,602 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257faa5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,605 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,605 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,605 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,606 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,609 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,609 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/012ce1b072ea46cf9fc0dfb03441493b 2023-04-20 20:55:09,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,618 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,619 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,719 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:09,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:09,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:09,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:09,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:09,722 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:09,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:09,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:09,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,723 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,723 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:09,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:09,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:09,725 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,725 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:09,725 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:09,725 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,727 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,727 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c653d8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,728 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,728 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,730 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,730 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,730 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,730 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,734 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,734 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:09,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:09,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,739 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,739 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,740 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024169837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:09,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024169837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,840 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:09,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:09,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:09,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:09,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:09,843 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:09,843 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,843 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:09,843 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:09,843 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,844 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,846 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,846 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,846 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,846 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:09,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:09,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:09,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:09,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:09,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:09,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:09,847 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,847 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:09,847 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:09,848 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,849 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66621e1c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,851 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,852 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,852 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,853 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,853 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,853 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,855 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,856 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,859 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:09,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,862 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,862 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,963 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:09,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:09,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:09,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:09,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:09,966 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:09,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:09,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:09,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:09,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:09,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:09,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:09,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:09,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:09,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:09,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:09,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:09,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:09,972 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,973 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:09,973 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:09,973 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:09,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:09,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:09,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:09,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:09,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,974 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:09,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3445126f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:09,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:09,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:09,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:09,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:09,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:09,980 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:09,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:09,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:09,982 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:09,982 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:09,982 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:09,982 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:09,982 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:09,984 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,984 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:09,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,984 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:09,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:09,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:09,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:09,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:09,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:09,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:09,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:09,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:09,992 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:09,991 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,991 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:09,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:09,992 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:09,992 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:09,992 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/9011562d736b4bc6acd0f91e915f80dd 2023-04-20 20:55:10,092 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:10,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:10,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:10,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:10,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:10,096 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:10,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:10,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:10,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:10,098 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:10,098 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,098 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:10,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:10,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:10,099 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:10,099 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,099 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:10,100 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,100 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:10,100 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:10,100 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:10,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,102 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,102 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,102 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:10,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97a4490[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:10,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:10,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,105 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:10,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:10,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:10,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:10,107 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:10,107 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:10,107 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:10,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:10,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:10,109 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:10,109 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:10,109 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:10,109 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:10,109 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:10,111 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,111 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,111 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:10,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:10,111 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,113 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,115 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:10,115 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:10,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:10,115 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:10,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:10,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:10,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:10,115 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,115 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,115 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,116 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,116 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,116 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,116 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:10,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:10,116 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:10,216 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:10,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:10,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:10,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:10,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:10,219 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:10,219 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,219 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:10,219 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:10,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,222 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:10,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:10,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,222 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:10,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:10,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:10,224 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,224 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:10,224 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:10,224 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:10,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,225 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:10,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:10,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f149ef4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:10,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:10,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:10,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:10,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:10,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:10,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:10,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:10,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:10,232 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:10,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:10,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:10,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:10,232 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:10,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:10,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,235 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,236 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:10,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:10,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:10,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,238 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:10,238 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:10,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:10,238 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:10,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,339 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:10,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:10,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:10,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024170339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:10,342 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:10,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:10,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:10,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,344 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:10,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:10,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:10,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:10,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:10,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024170343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:10,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:10,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:10,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:10,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:10,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:10,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4970e5f4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:10,350 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:10,350 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:10,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:10,353 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:10,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:10,355 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:10,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:10,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:10,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:10,355 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:10,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:10,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:10,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:10,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:10,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:10,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:10,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:10,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:10,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:10,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:10,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:10,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:10,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,361 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:10,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,361 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:10,362 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:10,362 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/3fea3cfc84c54c51aed236d93f47e3f6 2023-04-20 20:55:10,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/012ce1b072ea46cf9fc0dfb03441493b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/012ce1b072ea46cf9fc0dfb03441493b 2023-04-20 20:55:10,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/012ce1b072ea46cf9fc0dfb03441493b, entries=150, sequenceid=709, filesize=11.9 K 2023-04-20 20:55:10,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/9011562d736b4bc6acd0f91e915f80dd as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9011562d736b4bc6acd0f91e915f80dd 2023-04-20 20:55:10,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9011562d736b4bc6acd0f91e915f80dd, entries=150, sequenceid=709, filesize=11.9 K 2023-04-20 20:55:10,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/3fea3cfc84c54c51aed236d93f47e3f6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/3fea3cfc84c54c51aed236d93f47e3f6 2023-04-20 20:55:10,462 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:10,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:10,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:10,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:10,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:10,465 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:10,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:10,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:10,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/3fea3cfc84c54c51aed236d93f47e3f6, entries=150, sequenceid=709, filesize=11.9 K 2023-04-20 20:55:10,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,466 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bd7cf796b1328b6fd145cda0a337b2e3 in 1259ms, sequenceid=709, compaction requested=true 2023-04-20 20:55:10,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:10,466 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:10,466 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:10,466 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:10,466 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:10,466 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:10,467 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:10,467 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:10,467 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:55:10,467 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:55:10,467 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,467 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,467 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/274e7805a32a48f9925e53098d61993f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9e4a161ae494488aa5e0964c450e917e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/012ce1b072ea46cf9fc0dfb03441493b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=36.5 K 2023-04-20 20:55:10,467 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c9222f5972de4819b6a936e859864504, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/583cc231d01845b2b5df92502fd8d76c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9011562d736b4bc6acd0f91e915f80dd] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=36.5 K 2023-04-20 20:55:10,468 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 274e7805a32a48f9925e53098d61993f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1682024107711 2023-04-20 20:55:10,468 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c9222f5972de4819b6a936e859864504, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1682024107711 2023-04-20 20:55:10,468 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9e4a161ae494488aa5e0964c450e917e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1682024107763 2023-04-20 20:55:10,468 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 583cc231d01845b2b5df92502fd8d76c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1682024107763 2023-04-20 20:55:10,468 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 012ce1b072ea46cf9fc0dfb03441493b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682024108891 2023-04-20 20:55:10,469 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9011562d736b4bc6acd0f91e915f80dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682024108891 2023-04-20 20:55:10,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,471 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:10,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:10,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:10,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:10,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:10,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:10,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:10,472 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:10,472 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:10,472 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:10,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:10,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:10,473 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,473 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:10,473 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 20:55:10,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:10,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:10,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:10,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:10,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:10,474 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:10,475 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:10,476 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:10,900 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/279be214110c4b7e85c9564485cc014b 2023-04-20 20:55:10,904 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/ffe56b638c19485e9e24b2720a6fe17c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ffe56b638c19485e9e24b2720a6fe17c 2023-04-20 20:55:10,904 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/5cae8158d0e4460f92659988e1544a09 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/5cae8158d0e4460f92659988e1544a09 2023-04-20 20:55:10,910 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into 5cae8158d0e4460f92659988e1544a09(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:10,910 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:10,910 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=13, startTime=1682024110466; duration=0sec 2023-04-20 20:55:10,910 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:10,910 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:10,911 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into ffe56b638c19485e9e24b2720a6fe17c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:10,911 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:10,911 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=13, startTime=1682024110466; duration=0sec 2023-04-20 20:55:10,911 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:10,911 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:10,911 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:55:10,911 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:10,911 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/15491159496842a0a422a86428efa0b2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/82e38fc393ea437a8cbdbe2f72369bdf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/3fea3cfc84c54c51aed236d93f47e3f6] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=36.5 K 2023-04-20 20:55:10,912 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 15491159496842a0a422a86428efa0b2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1682024107711 2023-04-20 20:55:10,916 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 82e38fc393ea437a8cbdbe2f72369bdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1682024107763 2023-04-20 20:55:10,916 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3fea3cfc84c54c51aed236d93f47e3f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682024108891 2023-04-20 20:55:10,923 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#264 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:10,931 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/46dd7bdd94094d6280489456b086e2db as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/46dd7bdd94094d6280489456b086e2db 2023-04-20 20:55:10,936 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into 46dd7bdd94094d6280489456b086e2db(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:10,936 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:10,936 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=13, startTime=1682024110466; duration=0sec 2023-04-20 20:55:10,936 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:11,318 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/9d0c2c34dbfa4fc285b613a84188984f 2023-04-20 20:55:11,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:11,350 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:11,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:11,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024171365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:11,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024171367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:11,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024171468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:11,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024171470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:11,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024171670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:11,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024171672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,730 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9 2023-04-20 20:55:11,734 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/279be214110c4b7e85c9564485cc014b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/279be214110c4b7e85c9564485cc014b 2023-04-20 20:55:11,737 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/279be214110c4b7e85c9564485cc014b, entries=150, sequenceid=726, filesize=11.9 K 2023-04-20 20:55:11,738 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/9d0c2c34dbfa4fc285b613a84188984f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9d0c2c34dbfa4fc285b613a84188984f 2023-04-20 20:55:11,742 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9d0c2c34dbfa4fc285b613a84188984f, entries=150, sequenceid=726, filesize=11.9 K 2023-04-20 20:55:11,742 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9 2023-04-20 20:55:11,745 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9, entries=150, sequenceid=726, filesize=11.9 K 2023-04-20 20:55:11,746 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bd7cf796b1328b6fd145cda0a337b2e3 in 1273ms, sequenceid=726, compaction requested=false 2023-04-20 20:55:11,746 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:11,746 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,747 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:11,747 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:11,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:11,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:11,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:11,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ba50df[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:11,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:11,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:11,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:11,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:11,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,752 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:11,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:11,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:11,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:11,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:11,755 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:11,755 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:11,755 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:11,755 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:11,755 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:11,758 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,758 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:11,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,758 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:11,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:11,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:11,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:11,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:11,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:11,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:11,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:11,769 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:11,769 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,769 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:11,869 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:11,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:11,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:11,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:11,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:11,874 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:11,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:11,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:11,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:11,875 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,875 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:11,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,876 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:11,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:11,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:11,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:11,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:11,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:11,878 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:11,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:11,886 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/87c4998e387d472fbc4df473da5fd4d7 2023-04-20 20:55:11,896 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/ce929479163943e5ad347ba4eb1954bb 2023-04-20 20:55:11,906 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f8f48d45d9b744c8b983b0e5009d794c 2023-04-20 20:55:11,910 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/87c4998e387d472fbc4df473da5fd4d7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/87c4998e387d472fbc4df473da5fd4d7 2023-04-20 20:55:11,913 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/87c4998e387d472fbc4df473da5fd4d7, entries=150, sequenceid=745, filesize=11.9 K 2023-04-20 20:55:11,914 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/ce929479163943e5ad347ba4eb1954bb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ce929479163943e5ad347ba4eb1954bb 2023-04-20 20:55:11,918 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ce929479163943e5ad347ba4eb1954bb, entries=150, sequenceid=745, filesize=11.9 K 2023-04-20 20:55:11,918 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f8f48d45d9b744c8b983b0e5009d794c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f8f48d45d9b744c8b983b0e5009d794c 2023-04-20 20:55:11,922 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f8f48d45d9b744c8b983b0e5009d794c, entries=150, sequenceid=745, filesize=11.9 K 2023-04-20 20:55:11,923 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for bd7cf796b1328b6fd145cda0a337b2e3 in 45ms, sequenceid=745, compaction requested=true 2023-04-20 20:55:11,923 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:11,923 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:11,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:11,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:11,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:11,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:11,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:11,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:11,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:11,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ebfc03c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:11,926 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:11,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,929 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:11,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:11,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:11,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:11,930 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:11,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,930 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:11,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:11,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:11,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:11,934 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:11,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:11,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:11,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:11,934 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:11,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:11,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:11,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:11,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:11,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:11,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:11,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:11,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:11,941 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:11,941 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:11,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:11,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:11,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:55:11,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:11,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:11,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:11,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:11,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:11,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:11,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=755 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/94e0b67075124047a4b52d201443beb4 2023-04-20 20:55:12,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=755 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/1c12732c0b0a4151a93ea70a1da02106 2023-04-20 20:55:12,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024112001, max timestamp=1682024112013Num uniques 859; 2023-04-20 20:55:12,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:12,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024112001, max timestamp=1682024112013Num uniques 859; 2023-04-20 20:55:12,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024112001, max timestamp=1682024112013Num uniques 859; 2023-04-20 20:55:12,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:12,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:12,041 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:12,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:12,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:12,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024172040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024172041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:12,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:12,045 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:12,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:12,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:12,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,046 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,048 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:12,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:12,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:12,049 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:12,049 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:12,050 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,050 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:12,050 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:12,050 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:12,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:12,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:12,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15471c21[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:12,053 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:12,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:12,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:12,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:12,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:12,056 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:12,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:12,059 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:12,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:12,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:12,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:12,059 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:12,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,062 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,063 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:12,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:12,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:12,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,066 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:12,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,066 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:12,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024172145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024172145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,166 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:12,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:12,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:12,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:12,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:12,170 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:12,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:12,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:12,171 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,171 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:12,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:12,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:12,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:12,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:12,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:12,174 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,174 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:12,174 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:12,174 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:12,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:12,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:12,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:12,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,179 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,179 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:12,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bd60438[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:12,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:12,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:12,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:12,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:12,187 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,187 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:12,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:12,190 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:12,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:12,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:12,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:12,190 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:12,193 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,193 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,193 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,194 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:12,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,198 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,198 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:12,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,198 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:12,198 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:12,298 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:12,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:12,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:12,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:12,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:12,301 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:12,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:12,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:12,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:12,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:12,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:12,305 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,305 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:12,305 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:12,305 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:12,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:12,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:12,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:12,306 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,306 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,306 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,307 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:12,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c6d8b35[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:12,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:12,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,315 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:12,315 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:12,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:12,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:12,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:12,319 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:12,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:12,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:12,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:12,319 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:12,320 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,320 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,320 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,321 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:12,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:12,324 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,324 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:12,324 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,325 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024172347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024172348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=755 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/ee3be8209f0548819ba31311919985f2 2023-04-20 20:55:12,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/94e0b67075124047a4b52d201443beb4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/94e0b67075124047a4b52d201443beb4 2023-04-20 20:55:12,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/94e0b67075124047a4b52d201443beb4, entries=150, sequenceid=755, filesize=11.9 K 2023-04-20 20:55:12,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/1c12732c0b0a4151a93ea70a1da02106 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/1c12732c0b0a4151a93ea70a1da02106 2023-04-20 20:55:12,424 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:12,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:12,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/1c12732c0b0a4151a93ea70a1da02106, entries=100, sequenceid=755, filesize=9.5 K 2023-04-20 20:55:12,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/ee3be8209f0548819ba31311919985f2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ee3be8209f0548819ba31311919985f2 2023-04-20 20:55:12,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:12,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:12,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:12,428 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:12,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:12,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:12,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,430 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:12,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:12,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:12,432 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,432 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:12,432 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:12,432 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,433 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:12,433 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:12,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ee3be8209f0548819ba31311919985f2, entries=100, sequenceid=755, filesize=9.5 K 2023-04-20 20:55:12,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bd7cf796b1328b6fd145cda0a337b2e3 in 452ms, sequenceid=755, compaction requested=true 2023-04-20 20:55:12,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:12,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:12,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:12,435 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:12,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 20:55:12,436 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49626 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:12,436 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:55:12,436 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,436 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ffe56b638c19485e9e24b2720a6fe17c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/279be214110c4b7e85c9564485cc014b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/87c4998e387d472fbc4df473da5fd4d7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/94e0b67075124047a4b52d201443beb4] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=48.5 K 2023-04-20 20:55:12,437 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ffe56b638c19485e9e24b2720a6fe17c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682024108891 2023-04-20 20:55:12,437 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 279be214110c4b7e85c9564485cc014b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1682024109221 2023-04-20 20:55:12,438 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:12,438 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 87c4998e387d472fbc4df473da5fd4d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682024111360 2023-04-20 20:55:12,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,438 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 94e0b67075124047a4b52d201443beb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1682024111979 2023-04-20 20:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c231ea4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:12,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:12,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,442 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,442 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:12,444 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:12,444 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:55:12,444 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,445 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/5cae8158d0e4460f92659988e1544a09, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9d0c2c34dbfa4fc285b613a84188984f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ce929479163943e5ad347ba4eb1954bb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/1c12732c0b0a4151a93ea70a1da02106] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=46.1 K 2023-04-20 20:55:12,445 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5cae8158d0e4460f92659988e1544a09, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682024108891 2023-04-20 20:55:12,446 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9d0c2c34dbfa4fc285b613a84188984f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1682024109221 2023-04-20 20:55:12,447 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ce929479163943e5ad347ba4eb1954bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682024111360 2023-04-20 20:55:12,447 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1c12732c0b0a4151a93ea70a1da02106, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1682024111979 2023-04-20 20:55:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:12,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:12,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:12,453 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:12,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:12,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:12,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:12,454 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:12,456 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,456 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,456 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,458 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#272 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,462 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:12,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,463 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:12,463 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,466 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:12,500 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/8986a9dc7b2d43aeb2d4d35cbed52417 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8986a9dc7b2d43aeb2d4d35cbed52417 2023-04-20 20:55:12,504 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into 8986a9dc7b2d43aeb2d4d35cbed52417(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:12,505 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:12,505 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=12, startTime=1682024112435; duration=0sec 2023-04-20 20:55:12,505 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:12,505 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:12,513 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:12,513 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:55:12,513 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,513 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/46dd7bdd94094d6280489456b086e2db, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f8f48d45d9b744c8b983b0e5009d794c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ee3be8209f0548819ba31311919985f2] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=46.1 K 2023-04-20 20:55:12,514 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 46dd7bdd94094d6280489456b086e2db, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1682024108891 2023-04-20 20:55:12,514 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting fd76d0c5f67345c3a9f8f14e1a5f2dc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1682024109221 2023-04-20 20:55:12,514 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f8f48d45d9b744c8b983b0e5009d794c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682024111360 2023-04-20 20:55:12,514 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ee3be8209f0548819ba31311919985f2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1682024111979 2023-04-20 20:55:12,521 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:12,529 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/327fc52b6b394964ac3beb95a807783c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/327fc52b6b394964ac3beb95a807783c 2023-04-20 20:55:12,533 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into 327fc52b6b394964ac3beb95a807783c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:12,534 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:12,534 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=12, startTime=1682024112435; duration=0sec 2023-04-20 20:55:12,534 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:12,563 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:12,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:12,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:12,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:12,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:12,567 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:12,567 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,567 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:12,567 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:12,567 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,567 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:12,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:12,570 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:12,570 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,570 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:12,570 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 20:55:12,571 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:12,571 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:12,571 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:12,571 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:12,571 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:12,571 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:12,585 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/c0faaa50ec7d4b25b3513398b2383993 2023-04-20 20:55:12,597 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/29d51fcaca01472fa82f14b11d548861 2023-04-20 20:55:12,606 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/d02caeda6dd34dff93ba7bcdddcd2fa7 2023-04-20 20:55:12,612 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/c0faaa50ec7d4b25b3513398b2383993 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c0faaa50ec7d4b25b3513398b2383993 2023-04-20 20:55:12,615 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c0faaa50ec7d4b25b3513398b2383993, entries=150, sequenceid=794, filesize=11.9 K 2023-04-20 20:55:12,616 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/29d51fcaca01472fa82f14b11d548861 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/29d51fcaca01472fa82f14b11d548861 2023-04-20 20:55:12,620 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/29d51fcaca01472fa82f14b11d548861, entries=150, sequenceid=794, filesize=11.9 K 2023-04-20 20:55:12,620 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/d02caeda6dd34dff93ba7bcdddcd2fa7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/d02caeda6dd34dff93ba7bcdddcd2fa7 2023-04-20 20:55:12,624 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/d02caeda6dd34dff93ba7bcdddcd2fa7, entries=150, sequenceid=794, filesize=11.9 K 2023-04-20 20:55:12,625 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for bd7cf796b1328b6fd145cda0a337b2e3 in 55ms, sequenceid=794, compaction requested=false 2023-04-20 20:55:12,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:12,625 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:12,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:12,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8b915[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:12,629 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:12,629 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,632 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:12,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:12,636 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,637 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:12,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:12,639 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:12,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:12,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:12,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:12,639 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:12,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,643 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,645 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:12,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,646 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:12,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:12,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,646 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:12,646 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:12,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:12,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:55:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:12,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:12,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:12,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:12,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024112696, max timestamp=1682024112706Num uniques 859; 2023-04-20 20:55:12,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:12,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024112696, max timestamp=1682024112706Num uniques 859; 2023-04-20 20:55:12,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:12,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024112696, max timestamp=1682024112706Num uniques 859; 2023-04-20 20:55:12,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:12,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024172737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024172740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,746 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:12,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:12,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:12,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:12,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:12,750 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:12,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:12,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:12,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,752 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,752 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:12,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:12,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:12,754 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,754 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:12,754 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:12,754 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:12,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:12,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a349adc[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:12,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:12,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:12,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:12,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:12,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:12,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:12,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:12,771 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:12,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:12,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:12,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:12,771 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:12,772 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,772 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,772 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,774 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,777 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,777 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,777 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,777 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,778 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:12,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:12,777 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:12,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,778 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,778 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,778 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,778 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:12,778 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:12,778 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024172841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:12,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024172843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,878 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:12,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:12,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:12,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:12,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:12,881 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:12,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:12,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:12,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,883 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,883 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,883 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,883 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:12,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:12,885 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,885 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:12,885 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:12,885 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:12,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:12,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:12,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a30908c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:12,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:12,889 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:12,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:12,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:12,890 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,890 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:12,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:12,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:12,895 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:12,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:12,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:12,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:12,895 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:12,897 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,897 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,897 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:12,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:12,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:12,899 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/90bd9ad9d3174973b944aa4e2ca85f32 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/90bd9ad9d3174973b944aa4e2ca85f32 2023-04-20 20:55:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:12,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:12,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:12,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:12,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:12,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:12,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:12,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:12,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:12,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:12,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:12,903 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:12,903 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:12,904 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into 90bd9ad9d3174973b944aa4e2ca85f32(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:12,904 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:12,904 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=12, startTime=1682024112435; duration=0sec 2023-04-20 20:55:12,904 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:13,003 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:13,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:13,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:13,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:13,007 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:13,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:13,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:13,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,008 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,008 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:13,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:13,010 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,010 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:13,010 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:13,010 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:13,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,013 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:13,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e526350[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:13,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:13,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:13,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:13,017 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:13,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:13,020 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:13,020 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:13,020 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:13,020 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:13,020 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:13,025 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,025 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,025 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,027 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:13,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:13,029 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,030 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:13,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,030 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:13,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:13,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024173045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:13,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024173045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/7a0501d5a30044b893605803d06779d9 2023-04-20 20:55:13,130 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:13,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:13,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:13,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:13,133 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:13,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:13,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:13,134 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,134 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,140 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:13,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:13,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:13,142 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,142 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:13,142 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:13,142 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:13,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:13,145 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:13,147 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:13,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2af61f9e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:13,147 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,148 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:13,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:13,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:13,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:13,150 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:13,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:13,152 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:13,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:13,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:13,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:13,152 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:13,155 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,155 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,155 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,155 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:13,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:13,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:13,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,159 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:13,159 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:13,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,260 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:13,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:13,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:13,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:13,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:13,262 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:13,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:13,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:13,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,266 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:13,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:13,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:13,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:13,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:13,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:13,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:13,268 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:13,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:13,269 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,269 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,269 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,269 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:13,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645a9266[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:13,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:13,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,272 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:13,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:13,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:13,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:13,276 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:13,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:13,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:13,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:13,276 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:13,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,279 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,280 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:13,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:13,283 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,283 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,283 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,283 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:13,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024173349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:13,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024173351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,383 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:13,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:13,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:13,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:13,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:13,386 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:13,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:13,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:13,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,388 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,388 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,388 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,388 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:13,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:13,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:13,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:13,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:13,390 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,390 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:13,390 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:13,390 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:13,391 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,391 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:13,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ab0ca91[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:13,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,396 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:13,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:13,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:13,397 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:13,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:13,399 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:13,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:13,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:13,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:13,399 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:13,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,409 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:13,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,409 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:13,410 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/a76f8b099906428a93e21f47843f3268 2023-04-20 20:55:13,510 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:13,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:13,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:13,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:13,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:13,512 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:13,512 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,512 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:13,512 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:13,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,514 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,514 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,514 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,514 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:13,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:13,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:13,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:13,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:13,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:13,516 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,516 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. started... 2023-04-20 20:55:13,516 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. as already flushing 2023-04-20 20:55:13,516 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:13,517 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,518 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,518 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47057674[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:13,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:13,520 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,521 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,521 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:13,521 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:13,521 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:13,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:13,524 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:13,524 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,524 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,524 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:13,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:13,526 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:13,526 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:13,526 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:13,526 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:13,527 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:13,528 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,528 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,528 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:13,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:13,530 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:13,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:13,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:13,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:13,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:13,532 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,532 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,532 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:13,532 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:13,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:13,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:13,532 DEBUG [Thread-1449] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:13,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:13,532 DEBUG [Thread-1449] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:13,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:13,623 DEBUG [Thread-1456] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:62055 2023-04-20 20:55:13,623 DEBUG [Thread-1456] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:13,624 DEBUG [Thread-1458] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:62055 2023-04-20 20:55:13,624 DEBUG [Thread-1458] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:13,624 DEBUG [Thread-1454] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:62055 2023-04-20 20:55:13,624 DEBUG [Thread-1454] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:13,626 DEBUG [Thread-1452] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:62055 2023-04-20 20:55:13,626 DEBUG [Thread-1452] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:13,627 DEBUG [Thread-1450] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:62055 2023-04-20 20:55:13,627 DEBUG [Thread-1450] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:13,632 DEBUG [Thread-1449] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:13,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:13,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:13,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57986 deadline: 1682024173854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:13,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58032 deadline: 1682024173855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:13,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/51cf4756934c449ea89997366f82cbc3 2023-04-20 20:55:13,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/7a0501d5a30044b893605803d06779d9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/7a0501d5a30044b893605803d06779d9 2023-04-20 20:55:13,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/7a0501d5a30044b893605803d06779d9, entries=200, sequenceid=805, filesize=14.3 K 2023-04-20 20:55:13,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/a76f8b099906428a93e21f47843f3268 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a76f8b099906428a93e21f47843f3268 2023-04-20 20:55:13,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a76f8b099906428a93e21f47843f3268, entries=150, sequenceid=805, filesize=11.9 K 2023-04-20 20:55:13,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/51cf4756934c449ea89997366f82cbc3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/51cf4756934c449ea89997366f82cbc3 2023-04-20 20:55:13,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/51cf4756934c449ea89997366f82cbc3, entries=150, sequenceid=805, filesize=11.9 K 2023-04-20 20:55:13,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bd7cf796b1328b6fd145cda0a337b2e3 in 1244ms, sequenceid=805, compaction requested=true 2023-04-20 20:55:13,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:13,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:13,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:13,903 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:13,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:13,903 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:13,903 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:13,903 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/A is initiating minor compaction (all files) 2023-04-20 20:55:13,903 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:13,903 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/A in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,904 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/B is initiating minor compaction (all files) 2023-04-20 20:55:13,904 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/90bd9ad9d3174973b944aa4e2ca85f32, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c0faaa50ec7d4b25b3513398b2383993, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/7a0501d5a30044b893605803d06779d9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=39.1 K 2023-04-20 20:55:13,904 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/B in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:13,904 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8986a9dc7b2d43aeb2d4d35cbed52417, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/29d51fcaca01472fa82f14b11d548861, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a76f8b099906428a93e21f47843f3268] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=36.7 K 2023-04-20 20:55:13,904 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 90bd9ad9d3174973b944aa4e2ca85f32, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1682024111360 2023-04-20 20:55:13,904 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8986a9dc7b2d43aeb2d4d35cbed52417, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1682024111360 2023-04-20 20:55:13,904 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting c0faaa50ec7d4b25b3513398b2383993, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1682024112031 2023-04-20 20:55:13,904 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 29d51fcaca01472fa82f14b11d548861, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1682024112031 2023-04-20 20:55:13,904 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 7a0501d5a30044b893605803d06779d9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682024112653 2023-04-20 20:55:13,904 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a76f8b099906428a93e21f47843f3268, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682024112653 2023-04-20 20:55:13,910 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#A#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:13,910 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#B#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:14,319 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/24296896d1374969aaa653e56069b463 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/24296896d1374969aaa653e56069b463 2023-04-20 20:55:14,319 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/56f6b99ec0084cf18bdaafdd27923eef as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/56f6b99ec0084cf18bdaafdd27923eef 2023-04-20 20:55:14,322 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/A of bd7cf796b1328b6fd145cda0a337b2e3 into 56f6b99ec0084cf18bdaafdd27923eef(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:14,322 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/B of bd7cf796b1328b6fd145cda0a337b2e3 into 24296896d1374969aaa653e56069b463(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:14,323 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:14,323 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:14,323 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/A, priority=13, startTime=1682024113903; duration=0sec 2023-04-20 20:55:14,323 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/B, priority=13, startTime=1682024113903; duration=0sec 2023-04-20 20:55:14,323 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:14,323 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:14,323 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:14,324 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:14,324 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): bd7cf796b1328b6fd145cda0a337b2e3/C is initiating minor compaction (all files) 2023-04-20 20:55:14,324 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bd7cf796b1328b6fd145cda0a337b2e3/C in TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:14,324 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/327fc52b6b394964ac3beb95a807783c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/d02caeda6dd34dff93ba7bcdddcd2fa7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/51cf4756934c449ea89997366f82cbc3] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp, totalSize=36.7 K 2023-04-20 20:55:14,324 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 327fc52b6b394964ac3beb95a807783c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1682024111360 2023-04-20 20:55:14,324 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d02caeda6dd34dff93ba7bcdddcd2fa7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1682024112031 2023-04-20 20:55:14,325 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 51cf4756934c449ea89997366f82cbc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682024112653 2023-04-20 20:55:14,329 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): bd7cf796b1328b6fd145cda0a337b2e3#C#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:14,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:14,457 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 20:55:14,457 DEBUG [Thread-1445] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:62055 2023-04-20 20:55:14,457 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:14,457 DEBUG [Thread-1445] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:14,457 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:14,457 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:14,457 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:14,457 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:14,457 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:14,486 DEBUG [Thread-1441] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:62055 2023-04-20 20:55:14,486 DEBUG [Thread-1441] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:14,502 DEBUG [Thread-1439] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:62055 2023-04-20 20:55:14,502 DEBUG [Thread-1439] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:14,737 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/cea26781c15a45d58288dba2c2455fe2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cea26781c15a45d58288dba2c2455fe2 2023-04-20 20:55:14,741 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bd7cf796b1328b6fd145cda0a337b2e3/C of bd7cf796b1328b6fd145cda0a337b2e3 into cea26781c15a45d58288dba2c2455fe2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:14,741 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:14,741 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3., storeName=bd7cf796b1328b6fd145cda0a337b2e3/C, priority=13, startTime=1682024113903; duration=0sec 2023-04-20 20:55:14,741 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:14,859 DEBUG [Thread-1447] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:62055 2023-04-20 20:55:14,859 DEBUG [Thread-1447] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:14,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/83992ce0dcca44439718c5884db2a466 2023-04-20 20:55:14,866 DEBUG [Thread-1443] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:62055 2023-04-20 20:55:14,866 DEBUG [Thread-1443] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 346 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 323 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 5363 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 5185 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 5035 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 5356 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 5169 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 20:55:14,866 INFO [Listener at localhost/45619] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 20:55:14,867 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:62055 2023-04-20 20:55:14,867 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:14,867 INFO [Listener at localhost/45619] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 20:55:14,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-20 20:55:14,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:14,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 20:55:14,871 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024114871"}]},"ts":"1682024114871"} 2023-04-20 20:55:14,872 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 20:55:14,874 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 20:55:14,875 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bd7cf796b1328b6fd145cda0a337b2e3, UNASSIGN}] 2023-04-20 20:55:14,876 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bd7cf796b1328b6fd145cda0a337b2e3, UNASSIGN 2023-04-20 20:55:14,877 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=bd7cf796b1328b6fd145cda0a337b2e3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:14,877 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024114877"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024114877"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024114877"}]},"ts":"1682024114877"} 2023-04-20 20:55:14,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:55:14,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 20:55:15,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:15,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bd7cf796b1328b6fd145cda0a337b2e3, disabling compactions & flushes 2023-04-20 20:55:15,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:15,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 20:55:15,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/a862f9a5b69640a6ae5bfa866bc23183 2023-04-20 20:55:15,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 20:55:15,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f188cfb834464849a65997f16e9a7434 2023-04-20 20:55:15,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/83992ce0dcca44439718c5884db2a466 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/83992ce0dcca44439718c5884db2a466 2023-04-20 20:55:15,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/83992ce0dcca44439718c5884db2a466, entries=150, sequenceid=845, filesize=11.9 K 2023-04-20 20:55:15,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/a862f9a5b69640a6ae5bfa866bc23183 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a862f9a5b69640a6ae5bfa866bc23183 2023-04-20 20:55:15,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a862f9a5b69640a6ae5bfa866bc23183, entries=150, sequenceid=845, filesize=11.9 K 2023-04-20 20:55:15,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/f188cfb834464849a65997f16e9a7434 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f188cfb834464849a65997f16e9a7434 2023-04-20 20:55:15,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f188cfb834464849a65997f16e9a7434, entries=150, sequenceid=845, filesize=11.9 K 2023-04-20 20:55:15,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=26.84 KB/27480 for bd7cf796b1328b6fd145cda0a337b2e3 in 1241ms, sequenceid=845, compaction requested=false 2023-04-20 20:55:15,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:15,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. after waiting 0 ms 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:15,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bd7cf796b1328b6fd145cda0a337b2e3 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=A 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=B 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bd7cf796b1328b6fd145cda0a337b2e3, store=C 2023-04-20 20:55:15,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:15,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 20:55:16,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/d8e4de3de8f9457f8ad6a04f665a081d 2023-04-20 20:55:16,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/82234a29b2ad4b539f393d4c8ebac0a3 2023-04-20 20:55:16,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=853 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/5745e2e2df994508b1d7781709c2e59a 2023-04-20 20:55:16,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/A/d8e4de3de8f9457f8ad6a04f665a081d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/d8e4de3de8f9457f8ad6a04f665a081d 2023-04-20 20:55:16,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/d8e4de3de8f9457f8ad6a04f665a081d, entries=50, sequenceid=853, filesize=7.1 K 2023-04-20 20:55:16,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/B/82234a29b2ad4b539f393d4c8ebac0a3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/82234a29b2ad4b539f393d4c8ebac0a3 2023-04-20 20:55:16,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/82234a29b2ad4b539f393d4c8ebac0a3, entries=50, sequenceid=853, filesize=7.1 K 2023-04-20 20:55:16,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/.tmp/C/5745e2e2df994508b1d7781709c2e59a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5745e2e2df994508b1d7781709c2e59a 2023-04-20 20:55:16,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5745e2e2df994508b1d7781709c2e59a, entries=50, sequenceid=853, filesize=7.1 K 2023-04-20 20:55:16,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for bd7cf796b1328b6fd145cda0a337b2e3 in 1241ms, sequenceid=853, compaction requested=true 2023-04-20 20:55:16,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ccd656976ca744aeb3a4361fda939708, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/664490db0a564cc691434a40f48e22bb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2e7d97b6ab564cd881972c3287bd0d10, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a6c088104cde482390ccc3424a8814da, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a214adce74f14f55aceaa1abfc9351f6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6af48c8daeec416a825b5b3705117d78, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f373d035e1bc4b4f8217f7be188254d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9c30096c1a9f491f9d0ade9f797915eb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/528470999c7c454b93076843eaeec0f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/60bd417bd26e487eb3ba33653e6d2a14, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2750598b893f4374b8ec14340fa50506, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9ff0150432ec42d19c0cfdd5abed8f6c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6adebec56b48432f8b1b83dfaa7bf753, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6ee534af9a7c43d68ee655da3ee9cce5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/dc2749ba1fa044af85ff74cd0a7e33d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f16dac3176ef45e6bc035ebeb959ba24, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8c94d54c60ab4ecda8a73e53bb25a05e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a00e767eb3fc4d9ebc93d43544080e5a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/e432427e33bb4859bff343b7de6b57fb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/34256c8cef144bdb8c24cf291de7b382, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c1a6dbca15064fbe8998fa36bbed3f47, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9d648b24d4084a06bea0e6317a2df088, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8f5dad54bcf84376a27b293b11c7fbef, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/274e7805a32a48f9925e53098d61993f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9e4a161ae494488aa5e0964c450e917e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ffe56b638c19485e9e24b2720a6fe17c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/012ce1b072ea46cf9fc0dfb03441493b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/279be214110c4b7e85c9564485cc014b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/87c4998e387d472fbc4df473da5fd4d7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/90bd9ad9d3174973b944aa4e2ca85f32, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/94e0b67075124047a4b52d201443beb4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c0faaa50ec7d4b25b3513398b2383993, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/7a0501d5a30044b893605803d06779d9] to archive 2023-04-20 20:55:16,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:16,963 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ccd656976ca744aeb3a4361fda939708 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ccd656976ca744aeb3a4361fda939708 2023-04-20 20:55:16,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/664490db0a564cc691434a40f48e22bb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/664490db0a564cc691434a40f48e22bb 2023-04-20 20:55:16,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2e7d97b6ab564cd881972c3287bd0d10 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2e7d97b6ab564cd881972c3287bd0d10 2023-04-20 20:55:16,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a6c088104cde482390ccc3424a8814da to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a6c088104cde482390ccc3424a8814da 2023-04-20 20:55:16,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a214adce74f14f55aceaa1abfc9351f6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a214adce74f14f55aceaa1abfc9351f6 2023-04-20 20:55:16,968 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6af48c8daeec416a825b5b3705117d78 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6af48c8daeec416a825b5b3705117d78 2023-04-20 20:55:16,968 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f373d035e1bc4b4f8217f7be188254d8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f373d035e1bc4b4f8217f7be188254d8 2023-04-20 20:55:16,969 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9c30096c1a9f491f9d0ade9f797915eb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9c30096c1a9f491f9d0ade9f797915eb 2023-04-20 20:55:16,970 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/528470999c7c454b93076843eaeec0f7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/528470999c7c454b93076843eaeec0f7 2023-04-20 20:55:16,971 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/60bd417bd26e487eb3ba33653e6d2a14 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/60bd417bd26e487eb3ba33653e6d2a14 2023-04-20 20:55:16,972 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2750598b893f4374b8ec14340fa50506 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/2750598b893f4374b8ec14340fa50506 2023-04-20 20:55:16,973 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9ff0150432ec42d19c0cfdd5abed8f6c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9ff0150432ec42d19c0cfdd5abed8f6c 2023-04-20 20:55:16,974 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6adebec56b48432f8b1b83dfaa7bf753 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6adebec56b48432f8b1b83dfaa7bf753 2023-04-20 20:55:16,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 20:55:16,975 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6ee534af9a7c43d68ee655da3ee9cce5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/6ee534af9a7c43d68ee655da3ee9cce5 2023-04-20 20:55:16,975 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/dc2749ba1fa044af85ff74cd0a7e33d8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/dc2749ba1fa044af85ff74cd0a7e33d8 2023-04-20 20:55:16,976 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f16dac3176ef45e6bc035ebeb959ba24 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/f16dac3176ef45e6bc035ebeb959ba24 2023-04-20 20:55:16,977 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8c94d54c60ab4ecda8a73e53bb25a05e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8c94d54c60ab4ecda8a73e53bb25a05e 2023-04-20 20:55:16,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a00e767eb3fc4d9ebc93d43544080e5a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/a00e767eb3fc4d9ebc93d43544080e5a 2023-04-20 20:55:16,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/e432427e33bb4859bff343b7de6b57fb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/e432427e33bb4859bff343b7de6b57fb 2023-04-20 20:55:16,980 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/34256c8cef144bdb8c24cf291de7b382 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/34256c8cef144bdb8c24cf291de7b382 2023-04-20 20:55:16,980 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c1a6dbca15064fbe8998fa36bbed3f47 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c1a6dbca15064fbe8998fa36bbed3f47 2023-04-20 20:55:16,981 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9d648b24d4084a06bea0e6317a2df088 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9d648b24d4084a06bea0e6317a2df088 2023-04-20 20:55:16,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8f5dad54bcf84376a27b293b11c7fbef to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/8f5dad54bcf84376a27b293b11c7fbef 2023-04-20 20:55:16,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/274e7805a32a48f9925e53098d61993f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/274e7805a32a48f9925e53098d61993f 2023-04-20 20:55:16,984 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9e4a161ae494488aa5e0964c450e917e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/9e4a161ae494488aa5e0964c450e917e 2023-04-20 20:55:16,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ffe56b638c19485e9e24b2720a6fe17c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/ffe56b638c19485e9e24b2720a6fe17c 2023-04-20 20:55:16,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/012ce1b072ea46cf9fc0dfb03441493b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/012ce1b072ea46cf9fc0dfb03441493b 2023-04-20 20:55:16,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/279be214110c4b7e85c9564485cc014b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/279be214110c4b7e85c9564485cc014b 2023-04-20 20:55:16,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/87c4998e387d472fbc4df473da5fd4d7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/87c4998e387d472fbc4df473da5fd4d7 2023-04-20 20:55:16,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/90bd9ad9d3174973b944aa4e2ca85f32 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/90bd9ad9d3174973b944aa4e2ca85f32 2023-04-20 20:55:16,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/94e0b67075124047a4b52d201443beb4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/94e0b67075124047a4b52d201443beb4 2023-04-20 20:55:16,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c0faaa50ec7d4b25b3513398b2383993 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/c0faaa50ec7d4b25b3513398b2383993 2023-04-20 20:55:16,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/7a0501d5a30044b893605803d06779d9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/7a0501d5a30044b893605803d06779d9 2023-04-20 20:55:16,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b31cf195360f42b4a5b84e348d054ca3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/2ad257c8acba462e8cf63f9762acb162, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/12913d44284b41c3a71f92ee3ceb00b5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/31738304aa254b0cbdb532b61edf184d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9c54dc252a274f37a822f85fb39033bb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8800571806e344c4ba5b5eaa2209aa0a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ac07bbf54d2945e49be14eb020e1975c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d73c2cbc89ee4a728c30dc59e0f99dd3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/281236c2db52483688e2b1506b1926b3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/0006b2d1ae9b49a6840dac80b555a67e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/081d59ddf5804adbbbf2ac5b5754bc2a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b2d87e76493d456c999d2ff81c83a7c7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ef51c91de2a14c88917e60be6cf33977, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c454eaaa176d4a908e3e4776a83d35ba, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/68f403d86fd84daea4a942d4ffca2d9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/41bd778891bd45cb86602fe286f59bd0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d1748f2b516945b79566079b3594f445, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/677ddd2d3bec4f268b2227b7d3fd9cdc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/86bfa37b4bbf494c9023358165bd4683, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/cbe21ef228584fcab15f9bddf7d19d5f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/570141df101a4b5e85725e7328f089b8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/48e91716a66142dc90ae308f054fa447, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c9222f5972de4819b6a936e859864504, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/99707c458550467792f36c9074d451d9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/583cc231d01845b2b5df92502fd8d76c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/5cae8158d0e4460f92659988e1544a09, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9011562d736b4bc6acd0f91e915f80dd, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9d0c2c34dbfa4fc285b613a84188984f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ce929479163943e5ad347ba4eb1954bb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8986a9dc7b2d43aeb2d4d35cbed52417, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/1c12732c0b0a4151a93ea70a1da02106, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/29d51fcaca01472fa82f14b11d548861, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a76f8b099906428a93e21f47843f3268] to archive 2023-04-20 20:55:16,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:16,996 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b31cf195360f42b4a5b84e348d054ca3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b31cf195360f42b4a5b84e348d054ca3 2023-04-20 20:55:16,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/2ad257c8acba462e8cf63f9762acb162 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/2ad257c8acba462e8cf63f9762acb162 2023-04-20 20:55:16,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/12913d44284b41c3a71f92ee3ceb00b5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/12913d44284b41c3a71f92ee3ceb00b5 2023-04-20 20:55:16,999 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/31738304aa254b0cbdb532b61edf184d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/31738304aa254b0cbdb532b61edf184d 2023-04-20 20:55:17,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9c54dc252a274f37a822f85fb39033bb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9c54dc252a274f37a822f85fb39033bb 2023-04-20 20:55:17,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8800571806e344c4ba5b5eaa2209aa0a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8800571806e344c4ba5b5eaa2209aa0a 2023-04-20 20:55:17,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ac07bbf54d2945e49be14eb020e1975c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ac07bbf54d2945e49be14eb020e1975c 2023-04-20 20:55:17,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d73c2cbc89ee4a728c30dc59e0f99dd3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d73c2cbc89ee4a728c30dc59e0f99dd3 2023-04-20 20:55:17,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/281236c2db52483688e2b1506b1926b3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/281236c2db52483688e2b1506b1926b3 2023-04-20 20:55:17,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/0006b2d1ae9b49a6840dac80b555a67e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/0006b2d1ae9b49a6840dac80b555a67e 2023-04-20 20:55:17,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/081d59ddf5804adbbbf2ac5b5754bc2a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/081d59ddf5804adbbbf2ac5b5754bc2a 2023-04-20 20:55:17,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b2d87e76493d456c999d2ff81c83a7c7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/b2d87e76493d456c999d2ff81c83a7c7 2023-04-20 20:55:17,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ef51c91de2a14c88917e60be6cf33977 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ef51c91de2a14c88917e60be6cf33977 2023-04-20 20:55:17,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c454eaaa176d4a908e3e4776a83d35ba to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c454eaaa176d4a908e3e4776a83d35ba 2023-04-20 20:55:17,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/68f403d86fd84daea4a942d4ffca2d9d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/68f403d86fd84daea4a942d4ffca2d9d 2023-04-20 20:55:17,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/41bd778891bd45cb86602fe286f59bd0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/41bd778891bd45cb86602fe286f59bd0 2023-04-20 20:55:17,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d1748f2b516945b79566079b3594f445 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/d1748f2b516945b79566079b3594f445 2023-04-20 20:55:17,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/677ddd2d3bec4f268b2227b7d3fd9cdc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/677ddd2d3bec4f268b2227b7d3fd9cdc 2023-04-20 20:55:17,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/86bfa37b4bbf494c9023358165bd4683 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/86bfa37b4bbf494c9023358165bd4683 2023-04-20 20:55:17,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/cbe21ef228584fcab15f9bddf7d19d5f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/cbe21ef228584fcab15f9bddf7d19d5f 2023-04-20 20:55:17,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/570141df101a4b5e85725e7328f089b8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/570141df101a4b5e85725e7328f089b8 2023-04-20 20:55:17,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/48e91716a66142dc90ae308f054fa447 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/48e91716a66142dc90ae308f054fa447 2023-04-20 20:55:17,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c9222f5972de4819b6a936e859864504 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/c9222f5972de4819b6a936e859864504 2023-04-20 20:55:17,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/99707c458550467792f36c9074d451d9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/99707c458550467792f36c9074d451d9 2023-04-20 20:55:17,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/583cc231d01845b2b5df92502fd8d76c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/583cc231d01845b2b5df92502fd8d76c 2023-04-20 20:55:17,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/5cae8158d0e4460f92659988e1544a09 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/5cae8158d0e4460f92659988e1544a09 2023-04-20 20:55:17,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9011562d736b4bc6acd0f91e915f80dd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9011562d736b4bc6acd0f91e915f80dd 2023-04-20 20:55:17,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9d0c2c34dbfa4fc285b613a84188984f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/9d0c2c34dbfa4fc285b613a84188984f 2023-04-20 20:55:17,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ce929479163943e5ad347ba4eb1954bb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/ce929479163943e5ad347ba4eb1954bb 2023-04-20 20:55:17,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8986a9dc7b2d43aeb2d4d35cbed52417 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/8986a9dc7b2d43aeb2d4d35cbed52417 2023-04-20 20:55:17,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/1c12732c0b0a4151a93ea70a1da02106 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/1c12732c0b0a4151a93ea70a1da02106 2023-04-20 20:55:17,023 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/29d51fcaca01472fa82f14b11d548861 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/29d51fcaca01472fa82f14b11d548861 2023-04-20 20:55:17,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a76f8b099906428a93e21f47843f3268 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a76f8b099906428a93e21f47843f3268 2023-04-20 20:55:17,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5f6177759224493784828f047a93654b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/9281a5a166cc416eb26b8ee47e153b63, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f5593dfd3adc4b59ab07cc8c33418e14, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cf7ea7c91d284a66ae8355f94956ca80, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/70beb237988346d79cf27097c00e1d26, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2d267226bb1e4af1a2b67749dca71c40, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/8d4cb9886c6542d88b196be126bad07f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b42de5a2dc914f22b6369c0c488db505, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/30abd2469e6e48ddb06ba3d801d46a60, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ddbf094abe364ac694de00ab0698cd7b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ec25627a390c40388d591033a1d05d12, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/590e1c0fcd694799bba9723205335f2e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/c41f53e77a7a4bcbb63048575f86c5e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1c6ac6c1673240dbb9cf01da52413b47, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e09a39bc480e41aa8898bb45ebe5c2af, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f85d60d9663649f7bdc6d3354ee36846, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/56fff4fd551240bd8ac2605ed85577e0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1960ffc4395e40f492f90daa717279e6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e96321c7bc9b44f4b7b8a5f3e25d7aad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/de105b2c90c545858a01391bd682e648, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b6ea27dfb44d4db098226895f5b25013, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2ef75740de31409fbf4aacbe6ba80ce4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/15491159496842a0a422a86428efa0b2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/66f57cca4c3c40abb577dc2b1f4a81e3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/82e38fc393ea437a8cbdbe2f72369bdf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/46dd7bdd94094d6280489456b086e2db, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/3fea3cfc84c54c51aed236d93f47e3f6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f8f48d45d9b744c8b983b0e5009d794c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/327fc52b6b394964ac3beb95a807783c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ee3be8209f0548819ba31311919985f2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/d02caeda6dd34dff93ba7bcdddcd2fa7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/51cf4756934c449ea89997366f82cbc3] to archive 2023-04-20 20:55:17,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:17,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5f6177759224493784828f047a93654b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5f6177759224493784828f047a93654b 2023-04-20 20:55:17,029 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/9281a5a166cc416eb26b8ee47e153b63 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/9281a5a166cc416eb26b8ee47e153b63 2023-04-20 20:55:17,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f5593dfd3adc4b59ab07cc8c33418e14 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f5593dfd3adc4b59ab07cc8c33418e14 2023-04-20 20:55:17,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cf7ea7c91d284a66ae8355f94956ca80 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cf7ea7c91d284a66ae8355f94956ca80 2023-04-20 20:55:17,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/70beb237988346d79cf27097c00e1d26 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/70beb237988346d79cf27097c00e1d26 2023-04-20 20:55:17,032 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2d267226bb1e4af1a2b67749dca71c40 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2d267226bb1e4af1a2b67749dca71c40 2023-04-20 20:55:17,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/8d4cb9886c6542d88b196be126bad07f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/8d4cb9886c6542d88b196be126bad07f 2023-04-20 20:55:17,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b42de5a2dc914f22b6369c0c488db505 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b42de5a2dc914f22b6369c0c488db505 2023-04-20 20:55:17,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/30abd2469e6e48ddb06ba3d801d46a60 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/30abd2469e6e48ddb06ba3d801d46a60 2023-04-20 20:55:17,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ddbf094abe364ac694de00ab0698cd7b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ddbf094abe364ac694de00ab0698cd7b 2023-04-20 20:55:17,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ec25627a390c40388d591033a1d05d12 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ec25627a390c40388d591033a1d05d12 2023-04-20 20:55:17,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/590e1c0fcd694799bba9723205335f2e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/590e1c0fcd694799bba9723205335f2e 2023-04-20 20:55:17,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/c41f53e77a7a4bcbb63048575f86c5e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/c41f53e77a7a4bcbb63048575f86c5e7 2023-04-20 20:55:17,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1c6ac6c1673240dbb9cf01da52413b47 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1c6ac6c1673240dbb9cf01da52413b47 2023-04-20 20:55:17,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e09a39bc480e41aa8898bb45ebe5c2af to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e09a39bc480e41aa8898bb45ebe5c2af 2023-04-20 20:55:17,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f85d60d9663649f7bdc6d3354ee36846 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f85d60d9663649f7bdc6d3354ee36846 2023-04-20 20:55:17,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/56fff4fd551240bd8ac2605ed85577e0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/56fff4fd551240bd8ac2605ed85577e0 2023-04-20 20:55:17,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1960ffc4395e40f492f90daa717279e6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/1960ffc4395e40f492f90daa717279e6 2023-04-20 20:55:17,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e96321c7bc9b44f4b7b8a5f3e25d7aad to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/e96321c7bc9b44f4b7b8a5f3e25d7aad 2023-04-20 20:55:17,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/de105b2c90c545858a01391bd682e648 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/de105b2c90c545858a01391bd682e648 2023-04-20 20:55:17,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b6ea27dfb44d4db098226895f5b25013 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/b6ea27dfb44d4db098226895f5b25013 2023-04-20 20:55:17,046 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2ef75740de31409fbf4aacbe6ba80ce4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/2ef75740de31409fbf4aacbe6ba80ce4 2023-04-20 20:55:17,047 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/15491159496842a0a422a86428efa0b2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/15491159496842a0a422a86428efa0b2 2023-04-20 20:55:17,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/66f57cca4c3c40abb577dc2b1f4a81e3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/66f57cca4c3c40abb577dc2b1f4a81e3 2023-04-20 20:55:17,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/82e38fc393ea437a8cbdbe2f72369bdf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/82e38fc393ea437a8cbdbe2f72369bdf 2023-04-20 20:55:17,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/46dd7bdd94094d6280489456b086e2db to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/46dd7bdd94094d6280489456b086e2db 2023-04-20 20:55:17,052 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/3fea3cfc84c54c51aed236d93f47e3f6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/3fea3cfc84c54c51aed236d93f47e3f6 2023-04-20 20:55:17,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/fd76d0c5f67345c3a9f8f14e1a5f2dc9 2023-04-20 20:55:17,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f8f48d45d9b744c8b983b0e5009d794c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f8f48d45d9b744c8b983b0e5009d794c 2023-04-20 20:55:17,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/327fc52b6b394964ac3beb95a807783c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/327fc52b6b394964ac3beb95a807783c 2023-04-20 20:55:17,056 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ee3be8209f0548819ba31311919985f2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/ee3be8209f0548819ba31311919985f2 2023-04-20 20:55:17,057 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/d02caeda6dd34dff93ba7bcdddcd2fa7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/d02caeda6dd34dff93ba7bcdddcd2fa7 2023-04-20 20:55:17,058 DEBUG [StoreCloser-TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/51cf4756934c449ea89997366f82cbc3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/51cf4756934c449ea89997366f82cbc3 2023-04-20 20:55:17,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/recovered.edits/856.seqid, newMaxSeqId=856, maxSeqId=1 2023-04-20 20:55:17,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3. 2023-04-20 20:55:17,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bd7cf796b1328b6fd145cda0a337b2e3: 2023-04-20 20:55:17,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:17,065 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=bd7cf796b1328b6fd145cda0a337b2e3, regionState=CLOSED 2023-04-20 20:55:17,065 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024117065"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024117065"}]},"ts":"1682024117065"} 2023-04-20 20:55:17,067 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-20 20:55:17,067 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure bd7cf796b1328b6fd145cda0a337b2e3, server=jenkins-hbase4.apache.org,39223,1682024020002 in 2.1880 sec 2023-04-20 20:55:17,068 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-20 20:55:17,068 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bd7cf796b1328b6fd145cda0a337b2e3, UNASSIGN in 2.1920 sec 2023-04-20 20:55:17,069 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024117069"}]},"ts":"1682024117069"} 2023-04-20 20:55:17,069 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 20:55:17,071 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 20:55:17,072 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2040 sec 2023-04-20 20:55:18,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 20:55:18,976 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-20 20:55:18,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-20 20:55:18,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:18,978 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:18,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-20 20:55:18,979 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:18,983 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:18,985 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/recovered.edits] 2023-04-20 20:55:18,987 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/56f6b99ec0084cf18bdaafdd27923eef to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/56f6b99ec0084cf18bdaafdd27923eef 2023-04-20 20:55:18,988 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/83992ce0dcca44439718c5884db2a466 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/83992ce0dcca44439718c5884db2a466 2023-04-20 20:55:18,989 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/d8e4de3de8f9457f8ad6a04f665a081d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/A/d8e4de3de8f9457f8ad6a04f665a081d 2023-04-20 20:55:18,991 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/24296896d1374969aaa653e56069b463 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/24296896d1374969aaa653e56069b463 2023-04-20 20:55:18,992 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/82234a29b2ad4b539f393d4c8ebac0a3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/82234a29b2ad4b539f393d4c8ebac0a3 2023-04-20 20:55:18,993 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a862f9a5b69640a6ae5bfa866bc23183 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/B/a862f9a5b69640a6ae5bfa866bc23183 2023-04-20 20:55:18,995 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5745e2e2df994508b1d7781709c2e59a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/5745e2e2df994508b1d7781709c2e59a 2023-04-20 20:55:18,996 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cea26781c15a45d58288dba2c2455fe2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/cea26781c15a45d58288dba2c2455fe2 2023-04-20 20:55:18,997 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f188cfb834464849a65997f16e9a7434 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/C/f188cfb834464849a65997f16e9a7434 2023-04-20 20:55:18,999 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/recovered.edits/856.seqid to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3/recovered.edits/856.seqid 2023-04-20 20:55:19,000 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/bd7cf796b1328b6fd145cda0a337b2e3 2023-04-20 20:55:19,000 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:55:19,002 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:19,003 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 20:55:19,004 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 20:55:19,005 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:19,005 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 20:55:19,005 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682024119005"}]},"ts":"9223372036854775807"} 2023-04-20 20:55:19,007 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 20:55:19,007 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bd7cf796b1328b6fd145cda0a337b2e3, NAME => 'TestAcidGuarantees,,1682024091453.bd7cf796b1328b6fd145cda0a337b2e3.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 20:55:19,007 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 20:55:19,007 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682024119007"}]},"ts":"9223372036854775807"} 2023-04-20 20:55:19,008 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 20:55:19,010 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:19,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-04-20 20:55:19,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-20 20:55:19,080 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-20 20:55:19,091 INFO [Listener at localhost/45619] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 225) - Thread LEAK? -, OpenFileDescriptor=562 (was 489) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=354 (was 436), ProcessCount=169 (was 169), AvailableMemoryMB=8657 (was 8703) 2023-04-20 20:55:19,102 INFO [Listener at localhost/45619] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=562, MaxFileDescriptor=60000, SystemLoadAverage=354, ProcessCount=169, AvailableMemoryMB=8657 2023-04-20 20:55:19,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:55:19,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:55:19,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:19,107 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 20:55:19,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-20 20:55:19,108 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 20:55:19,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 20:55:19,109 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,109 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 empty. 2023-04-20 20:55:19,109 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,109 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:55:19,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 20:55:19,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 20:55:19,516 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:55:19,518 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7dcdfe0ddb3c517a4a0d9b1a81351709, NAME => 'TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp 2023-04-20 20:55:19,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:55:19,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7dcdfe0ddb3c517a4a0d9b1a81351709, disabling compactions & flushes 2023-04-20 20:55:19,524 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. after waiting 0 ms 2023-04-20 20:55:19,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,524 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:19,527 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 20:55:19,528 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024119528"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024119528"}]},"ts":"1682024119528"} 2023-04-20 20:55:19,529 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 20:55:19,530 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 20:55:19,530 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024119530"}]},"ts":"1682024119530"} 2023-04-20 20:55:19,531 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 20:55:19,536 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, ASSIGN}] 2023-04-20 20:55:19,537 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, ASSIGN 2023-04-20 20:55:19,538 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=false 2023-04-20 20:55:19,689 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:19,689 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024119689"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024119689"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024119689"}]},"ts":"1682024119689"} 2023-04-20 20:55:19,691 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:55:19,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 20:55:19,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7dcdfe0ddb3c517a4a0d9b1a81351709, NAME => 'TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:55:19,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:55:19,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,847 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,848 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A 2023-04-20 20:55:19,848 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A 2023-04-20 20:55:19,848 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:19,848 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7dcdfe0ddb3c517a4a0d9b1a81351709 columnFamilyName A 2023-04-20 20:55:19,849 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(310): Store=7dcdfe0ddb3c517a4a0d9b1a81351709/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:19,849 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,850 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B 2023-04-20 20:55:19,850 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B 2023-04-20 20:55:19,850 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:19,850 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7dcdfe0ddb3c517a4a0d9b1a81351709 columnFamilyName B 2023-04-20 20:55:19,851 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(310): Store=7dcdfe0ddb3c517a4a0d9b1a81351709/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:19,851 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,851 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C 2023-04-20 20:55:19,851 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C 2023-04-20 20:55:19,852 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:19,852 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7dcdfe0ddb3c517a4a0d9b1a81351709 columnFamilyName C 2023-04-20 20:55:19,852 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(310): Store=7dcdfe0ddb3c517a4a0d9b1a81351709/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:19,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:55:19,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:19,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:55:19,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7dcdfe0ddb3c517a4a0d9b1a81351709; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62195908, jitterRate=-0.073208749294281}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:55:19,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:19,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., pid=37, masterSystemTime=1682024119842 2023-04-20 20:55:19,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:19,859 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:19,859 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024119859"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024119859"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024119859"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024119859"}]},"ts":"1682024119859"} 2023-04-20 20:55:19,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-20 20:55:19,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 in 170 msec 2023-04-20 20:55:19,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-20 20:55:19,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, ASSIGN in 326 msec 2023-04-20 20:55:19,863 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 20:55:19,863 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024119863"}]},"ts":"1682024119863"} 2023-04-20 20:55:19,864 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 20:55:19,865 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 20:55:19,866 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 760 msec 2023-04-20 20:55:20,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 20:55:20,212 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-20 20:55:20,212 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:20,217 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:20,218 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:20,220 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:20,221 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 20:55:20,222 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39596, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 20:55:20,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:55:20,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:55:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:20,634 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 20:55:20,635 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:55:20,635 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 20:55:20,638 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-20 20:55:20,641 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, REOPEN/MOVE}] 2023-04-20 20:55:20,642 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, REOPEN/MOVE 2023-04-20 20:55:20,642 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:20,643 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024120642"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024120642"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024120642"}]},"ts":"1682024120642"} 2023-04-20 20:55:20,644 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:55:20,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:20,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7dcdfe0ddb3c517a4a0d9b1a81351709, disabling compactions & flushes 2023-04-20 20:55:20,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:20,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:20,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. after waiting 0 ms 2023-04-20 20:55:20,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:20,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-20 20:55:20,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:20,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:20,801 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 7dcdfe0ddb3c517a4a0d9b1a81351709 to self. 2023-04-20 20:55:20,802 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:20,803 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=CLOSED 2023-04-20 20:55:20,803 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024120803"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024120803"}]},"ts":"1682024120803"} 2023-04-20 20:55:20,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-20 20:55:20,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 in 160 msec 2023-04-20 20:55:20,806 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=true 2023-04-20 20:55:20,956 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:20,956 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024120956"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024120956"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024120956"}]},"ts":"1682024120956"} 2023-04-20 20:55:20,958 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:55:21,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7dcdfe0ddb3c517a4a0d9b1a81351709, NAME => 'TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:55:21,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:55:21,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,115 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,115 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A 2023-04-20 20:55:21,115 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A 2023-04-20 20:55:21,116 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:21,116 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7dcdfe0ddb3c517a4a0d9b1a81351709 columnFamilyName A 2023-04-20 20:55:21,118 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(310): Store=7dcdfe0ddb3c517a4a0d9b1a81351709/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:21,118 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,119 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B 2023-04-20 20:55:21,119 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B 2023-04-20 20:55:21,119 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:21,119 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7dcdfe0ddb3c517a4a0d9b1a81351709 columnFamilyName B 2023-04-20 20:55:21,120 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(310): Store=7dcdfe0ddb3c517a4a0d9b1a81351709/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:21,120 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,121 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C 2023-04-20 20:55:21,121 DEBUG [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C 2023-04-20 20:55:21,121 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:21,121 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7dcdfe0ddb3c517a4a0d9b1a81351709 columnFamilyName C 2023-04-20 20:55:21,122 INFO [StoreOpener-7dcdfe0ddb3c517a4a0d9b1a81351709-1] regionserver.HStore(310): Store=7dcdfe0ddb3c517a4a0d9b1a81351709/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:21,122 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:55:21,127 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7dcdfe0ddb3c517a4a0d9b1a81351709; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62826636, jitterRate=-0.06381016969680786}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:55:21,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:21,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., pid=42, masterSystemTime=1682024121109 2023-04-20 20:55:21,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,130 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,130 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024121130"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024121130"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024121130"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024121130"}]},"ts":"1682024121130"} 2023-04-20 20:55:21,133 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-20 20:55:21,133 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 in 173 msec 2023-04-20 20:55:21,134 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-20 20:55:21,134 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, REOPEN/MOVE in 492 msec 2023-04-20 20:55:21,136 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-20 20:55:21,136 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 497 msec 2023-04-20 20:55:21,138 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 913 msec 2023-04-20 20:55:21,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-20 20:55:21,140 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,145 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,145 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,149 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,149 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,155 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,156 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,162 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,162 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,167 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,167 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,180 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,181 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,192 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,192 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,204 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,204 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,210 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,211 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x0eb8f142 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:21,214 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e079da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:21,224 DEBUG [hconnection-0x2c7feb9e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,224 DEBUG [hconnection-0x6af5646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,226 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,229 DEBUG [hconnection-0x5f13ebed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,230 DEBUG [hconnection-0x1adeda5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,230 DEBUG [hconnection-0x3cf57fcf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,230 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,230 DEBUG [hconnection-0x14e1631d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,230 DEBUG [hconnection-0x3cb255ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,233 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,237 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,237 DEBUG [hconnection-0x38368c32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,238 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,244 DEBUG [hconnection-0x7f9fd71f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,244 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,244 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,245 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,245 DEBUG [hconnection-0x30cb2720-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:21,253 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,254 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:21,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:21,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:21,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:55:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:21,261 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:21,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:21,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:21,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:21,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:21,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:21,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:21,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:21,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:21,266 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,267 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,267 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:21,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:21,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:21,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:21,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:21,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:21,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024181279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024181280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024181280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024181280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024181281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:21,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,294 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:21,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d159be5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:21,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:21,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:21,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:21,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:21,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:21,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:21,301 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:21,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:21,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:21,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:21,301 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:21,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:21,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:21,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:21,312 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:21,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,312 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:21,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024181381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024181384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024181384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024181384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024181384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,413 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:21,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:21,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:21,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:21,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:21,429 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:21,429 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,429 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:21,429 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:21,429 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,429 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,431 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:21,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:21,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:21,432 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,432 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:21,433 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:21,433 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,433 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:21,433 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:21,433 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,433 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:21,433 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:21,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:21,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6decfb83[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:21,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:21,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:21,438 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,438 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:21,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:21,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:21,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:21,444 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:21,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:21,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:21,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:21,444 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:21,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,446 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,450 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:21,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,450 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:21,450 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:21,551 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:21,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:21,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:21,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:21,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:21,575 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:21,575 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,575 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:21,575 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:21,576 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,576 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,577 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,577 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,577 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,577 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:21,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024181582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:21,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:21,584 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,584 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:21,585 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:21,585 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:21,587 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513450ed[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:21,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:21,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,591 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:21,591 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:21,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:21,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,592 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024181586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024181586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,593 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,593 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024181587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024181587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:21,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:21,595 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:21,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:21,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:21,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:21,595 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:21,597 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,597 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,597 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,601 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:21,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:21,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:21,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,605 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:21,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,605 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:21,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,605 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,693 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420157b11c0e9ec454fb2e558e4169a0671 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420157b11c0e9ec454fb2e558e4169a0671 2023-04-20 20:55:21,705 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:21,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:21,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:21,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:21,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:21,709 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:21,709 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,709 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:21,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:21,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,711 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:21,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:21,713 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,713 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:21,713 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:21,713 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:21,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dee3a69[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:21,717 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:21,718 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,719 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:21,719 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,719 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:21,719 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:21,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,721 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:21,724 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:21,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:21,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:21,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:21,724 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:21,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,726 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,728 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:21,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,733 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,733 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,733 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:21,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,834 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:21,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:21,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:21,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:21,838 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:21,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:21,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:21,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,840 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,841 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,841 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:21,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:21,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:21,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:21,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,845 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:21,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd54482[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:21,846 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:21,847 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,848 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:21,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,848 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:21,850 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:21,853 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:21,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:21,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:21,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:21,853 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:21,855 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,855 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,855 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,858 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:21,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,862 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:21,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,862 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:21,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024181885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024181895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024181895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024181895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024181896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,962 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:21,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:21,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:21,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:21,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:21,965 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:21,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:21,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:21,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:21,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:21,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:21,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:21,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:21,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:21,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:21,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,971 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,971 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,971 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:21,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ef8e78[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:21,973 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:21,973 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:21,981 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,981 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:21,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:21,985 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:21,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:21,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:21,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:21,986 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:21,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:21,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:21,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,989 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,990 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:21,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:21,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:21,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:21,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,993 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:21,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:21,993 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:21,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,093 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:22,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:22,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:22,104 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:22,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:22,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:22,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,108 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,108 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,108 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,111 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/43f6a14ceb4047a1a65e58cfcdc2d491 2023-04-20 20:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:22,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:22,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:22,112 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,112 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:22,112 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:22,112 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:22,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:22,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:22,112 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:22,114 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,115 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,115 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:22,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35af1d70[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:22,117 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:22,117 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,118 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:22,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,118 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:22,118 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:22,118 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:22,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,120 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,120 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,120 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:22,124 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:22,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:22,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:22,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:22,124 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:22,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,127 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,129 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,131 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:22,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:22,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:22,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,132 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,132 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:22,132 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,132 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,132 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,132 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,232 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:22,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:22,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:22,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:22,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:22,236 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:22,236 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,236 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:22,236 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:22,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,241 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,242 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,242 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,242 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:22,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:22,248 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:22,248 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:22,248 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,248 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:22,248 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:22,248 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:22,252 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79de54de[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:22,257 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:22,257 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,258 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:22,259 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,259 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:22,263 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:22,265 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:22,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:22,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:22,266 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:22,266 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:22,269 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,269 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,270 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,270 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,275 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:22,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,275 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,275 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:22,375 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:22,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:22,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:22,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:22,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:22,379 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:22,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:22,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:22,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,380 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,380 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:22,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:22,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:22,382 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,382 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:22,382 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:22,382 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:22,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:22,384 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,384 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,384 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,384 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:22,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f49fe33[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:22,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:22,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,387 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:22,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:22,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:22,389 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024182390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:22,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:22,392 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:22,392 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:22,392 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:22,392 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:22,392 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:22,394 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,394 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,394 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,394 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,398 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,398 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,398 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,398 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,398 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:22,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:22,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,399 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:22,399 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:22,399 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,399 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,399 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,399 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,399 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:22,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024182400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:22,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024182402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:22,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024182403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:22,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024182406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,499 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:22,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:22,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:22,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:22,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:22,502 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:22,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:22,503 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:22,503 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,503 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:22,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:22,506 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,506 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:22,506 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:22,506 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:22,509 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:22,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@256f2fec[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:22,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,514 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:22,515 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,515 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:22,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:22,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:22,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:22,518 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:22,518 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:22,518 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:22,519 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:22,519 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:22,520 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,520 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,520 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:22,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,522 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/0b51b2b02a1d42539ee501fdadd18599 2023-04-20 20:55:22,525 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,525 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:22,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:22,525 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,525 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,525 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,525 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,525 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:22,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:22,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:22,525 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,526 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:22,526 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,526 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:22,526 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:22,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:22,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/b52e91fec7154df6aec60019bd0532c2 2023-04-20 20:55:22,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/43f6a14ceb4047a1a65e58cfcdc2d491 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/43f6a14ceb4047a1a65e58cfcdc2d491 2023-04-20 20:55:22,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/43f6a14ceb4047a1a65e58cfcdc2d491, entries=250, sequenceid=18, filesize=38.7 K 2023-04-20 20:55:22,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/0b51b2b02a1d42539ee501fdadd18599 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0b51b2b02a1d42539ee501fdadd18599 2023-04-20 20:55:22,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0b51b2b02a1d42539ee501fdadd18599, entries=150, sequenceid=18, filesize=11.6 K 2023-04-20 20:55:22,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/b52e91fec7154df6aec60019bd0532c2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/b52e91fec7154df6aec60019bd0532c2 2023-04-20 20:55:22,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/b52e91fec7154df6aec60019bd0532c2, entries=150, sequenceid=18, filesize=11.6 K 2023-04-20 20:55:22,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1298ms, sequenceid=18, compaction requested=false 2023-04-20 20:55:22,559 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-20 20:55:22,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:22,604 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 20:55:22,626 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:22,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:22,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:22,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:22,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:22,636 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:22,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:22,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:22,637 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,637 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,638 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:22,638 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:22,638 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:22,638 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:22,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:22,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:22,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:22,640 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:22,640 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:22,640 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:22,641 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:22,641 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:22,641 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:22,641 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:22,641 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 20:55:22,641 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:22,641 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:22,642 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:22,642 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:22,642 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:22,642 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:22,662 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304208dd371eb15f648a390c0bb47d3704155 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208dd371eb15f648a390c0bb47d3704155 2023-04-20 20:55:23,073 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/afe9db1575064e8883510fc357ecf47b 2023-04-20 20:55:23,082 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/2a86c048be474a5eb11544abeb7341d2 2023-04-20 20:55:23,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:23,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:23,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024183415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024183416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024183417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024183418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024183418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,496 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/05f2dfb8a1c44eb293ae7e9230c9f9aa 2023-04-20 20:55:23,501 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/afe9db1575064e8883510fc357ecf47b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/afe9db1575064e8883510fc357ecf47b 2023-04-20 20:55:23,504 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/afe9db1575064e8883510fc357ecf47b, entries=150, sequenceid=38, filesize=25.1 K 2023-04-20 20:55:23,505 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/2a86c048be474a5eb11544abeb7341d2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2a86c048be474a5eb11544abeb7341d2 2023-04-20 20:55:23,510 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2a86c048be474a5eb11544abeb7341d2, entries=150, sequenceid=38, filesize=11.6 K 2023-04-20 20:55:23,511 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/05f2dfb8a1c44eb293ae7e9230c9f9aa as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/05f2dfb8a1c44eb293ae7e9230c9f9aa 2023-04-20 20:55:23,514 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/05f2dfb8a1c44eb293ae7e9230c9f9aa, entries=150, sequenceid=38, filesize=11.6 K 2023-04-20 20:55:23,514 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 873ms, sequenceid=38, compaction requested=false 2023-04-20 20:55:23,514 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:23,514 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:23,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6866e2ac[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:23,518 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:23,518 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:23,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 20:55:23,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:23,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:23,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:23,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:23,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:23,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:23,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:23,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:23,530 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:23,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:23,534 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:23,535 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:23,535 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:23,535 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:23,535 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:23,536 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,537 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,537 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:23,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:23,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:23,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:23,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:23,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,546 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:23,546 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:23,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024183547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024183548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024183549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024183550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024183550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,646 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:23,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:23,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:23,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:23,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:23,650 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,653 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:23,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:23,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:23,654 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:23,654 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:23,655 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:23,655 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:23,655 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:23,655 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:23,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024183653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024183653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:23,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:23,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc672e1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:23,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:23,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:23,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,658 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:23,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:23,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:23,659 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,659 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:23,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:23,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024183657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:23,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:23,663 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:23,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:23,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:23,663 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:23,663 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:23,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024183658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024183658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,666 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,666 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,666 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,666 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,671 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:23,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:23,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:23,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:23,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:23,671 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:23,671 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:23,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,772 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:23,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:23,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:23,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:23,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:23,775 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:23,775 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:23,775 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:23,775 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:23,775 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,775 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,777 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:23,777 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:23,777 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,777 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:23,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:23,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:23,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:23,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:23,778 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:23,779 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:23,779 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:23,779 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:23,779 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:23,780 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,780 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:23,782 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:23,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@639523da[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:23,782 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,784 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,785 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:23,785 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:23,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:23,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:23,788 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:23,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:23,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:23,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:23,789 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:23,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,791 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:23,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:23,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:23,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:23,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,795 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:23,795 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:23,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024183857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024183859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024183864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024183864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:23,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024183865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,895 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:23,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:23,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:23,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:23,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:23,898 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:23,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:23,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:23,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:23,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:23,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:23,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:23,903 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:23,903 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:23,903 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:23,903 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:23,904 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337df1fd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:23,906 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:23,906 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,907 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:23,907 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:23,908 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:23,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:23,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:23,915 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:23,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:23,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:23,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:23,915 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:23,917 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,917 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:23,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,917 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:23,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,921 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,921 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,921 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:23,921 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:23,921 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:23,921 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:23,922 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:23,922 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:23,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:23,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:23,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:23,922 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:23,922 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:23,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:23,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:23,922 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:23,922 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:23,944 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420fbed9e9d20ed4068bf59f8181dbfef9a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fbed9e9d20ed4068bf59f8181dbfef9a 2023-04-20 20:55:23,950 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=53, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/6ded95415f1a442db0c999a9a1af5df9 2023-04-20 20:55:24,022 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:24,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:24,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:24,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:24,026 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,026 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:24,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:24,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:24,030 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,030 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:24,030 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:24,030 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,033 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:24,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb1a831[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:24,035 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:24,035 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,036 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,037 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:24,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:24,040 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:24,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:24,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:24,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:24,040 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:24,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,043 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,046 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:24,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:24,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:24,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,047 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:24,047 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:24,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,147 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:24,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:24,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:24,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:24,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:24,156 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:24,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:24,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:24,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,158 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:24,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:24,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:24,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:24,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:24,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:24,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:24,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:24,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024184162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024184163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:24,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301d44e9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:24,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:24,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,174 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:24,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:24,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:24,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024184170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024184171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024184171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:24,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:24,181 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:24,181 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:24,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:24,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:24,182 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:24,183 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,183 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,184 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,187 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:24,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,187 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:24,188 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:24,288 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:24,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:24,291 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:24,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:24,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:24,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,292 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:24,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:24,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:24,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:24,294 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,294 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:24,294 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:24,294 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:24,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,296 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a5c3ac3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:24,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:24,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:24,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:24,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:24,306 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:24,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:24,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:24,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:24,306 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:24,309 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,309 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,309 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:24,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:24,315 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,316 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:24,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:24,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,316 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:24,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,316 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/7a005429fb1949c4a879ce6272ffeb03 2023-04-20 20:55:24,416 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:24,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:24,420 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:24,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:24,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:24,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,422 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:24,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:24,424 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,424 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:24,424 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:24,424 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:24,426 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:24,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2672ce6c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:24,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:24,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,429 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:24,430 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,430 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:24,432 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:24,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,432 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:24,434 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:24,434 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:24,434 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:24,434 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:24,434 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:24,436 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,436 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,436 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,438 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:24,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:24,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:24,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,441 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:24,441 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:24,541 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:24,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:24,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:24,545 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:24,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:24,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:24,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:24,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:24,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:24,549 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,549 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:24,549 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:24,549 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:24,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,553 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e01ab3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:24,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:24,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,556 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:24,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:24,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:24,566 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:24,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:24,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:24,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:24,566 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:24,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:24,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:24,573 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,574 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:24,574 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:24,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,574 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,674 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:24,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024184671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:24,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024184673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:24,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:24,680 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:24,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:24,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:24,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:24,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:24,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:24,684 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,684 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:24,684 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:24,684 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:24,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024184676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024184681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:24,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024184681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0024ca[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:24,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:24,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:24,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:24,692 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:24,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:24,699 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:24,699 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:24,699 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:24,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:24,700 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:24,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:24,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,714 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:24,714 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:24,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:24,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/a085f0ff2a084ea4a15898f0903f4f51 2023-04-20 20:55:24,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/6ded95415f1a442db0c999a9a1af5df9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6ded95415f1a442db0c999a9a1af5df9 2023-04-20 20:55:24,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6ded95415f1a442db0c999a9a1af5df9, entries=200, sequenceid=53, filesize=31.9 K 2023-04-20 20:55:24,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/7a005429fb1949c4a879ce6272ffeb03 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a005429fb1949c4a879ce6272ffeb03 2023-04-20 20:55:24,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a005429fb1949c4a879ce6272ffeb03, entries=150, sequenceid=53, filesize=11.6 K 2023-04-20 20:55:24,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/a085f0ff2a084ea4a15898f0903f4f51 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a085f0ff2a084ea4a15898f0903f4f51 2023-04-20 20:55:24,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a085f0ff2a084ea4a15898f0903f4f51, entries=150, sequenceid=53, filesize=11.6 K 2023-04-20 20:55:24,796 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1269ms, sequenceid=53, compaction requested=true 2023-04-20 20:55:24,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:24,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:24,796 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:24,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:24,796 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:24,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:24,797 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:24,797 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:24,797 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:24,797 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,797 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:24,797 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/43f6a14ceb4047a1a65e58cfcdc2d491, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/afe9db1575064e8883510fc357ecf47b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6ded95415f1a442db0c999a9a1af5df9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=95.8 K 2023-04-20 20:55:24,797 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,797 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0b51b2b02a1d42539ee501fdadd18599, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2a86c048be474a5eb11544abeb7341d2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a005429fb1949c4a879ce6272ffeb03] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=34.8 K 2023-04-20 20:55:24,798 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 43f6a14ceb4047a1a65e58cfcdc2d491, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682024121256 2023-04-20 20:55:24,798 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0b51b2b02a1d42539ee501fdadd18599, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682024121257 2023-04-20 20:55:24,798 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting afe9db1575064e8883510fc357ecf47b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682024121277 2023-04-20 20:55:24,798 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2a86c048be474a5eb11544abeb7341d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682024121277 2023-04-20 20:55:24,799 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6ded95415f1a442db0c999a9a1af5df9, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682024123417 2023-04-20 20:55:24,799 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7a005429fb1949c4a879ce6272ffeb03, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682024123417 2023-04-20 20:55:24,805 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#300 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:24,810 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#299 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:24,814 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:24,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:24,817 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:24,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:24,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:24,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,819 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,820 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,820 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:24,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:24,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:24,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:24,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:24,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:24,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:24,821 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,821 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:24,821 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:55:24,822 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:24,822 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:24,822 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:24,822 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:24,822 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:24,822 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:24,843 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304203cf142af191440639c825af0e6fbad91 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203cf142af191440639c825af0e6fbad91 2023-04-20 20:55:24,849 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/4b5c34f679d745d8b91257e8df539db9 2023-04-20 20:55:24,865 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/9fef4a1c02eb4e70b4d4ea1f03310385 2023-04-20 20:55:24,878 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/1773933967db4efcbe8f3bf90db89a8b 2023-04-20 20:55:24,882 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/4b5c34f679d745d8b91257e8df539db9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/4b5c34f679d745d8b91257e8df539db9 2023-04-20 20:55:24,887 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/4b5c34f679d745d8b91257e8df539db9, entries=150, sequenceid=70, filesize=25.1 K 2023-04-20 20:55:24,888 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/9fef4a1c02eb4e70b4d4ea1f03310385 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/9fef4a1c02eb4e70b4d4ea1f03310385 2023-04-20 20:55:24,898 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/9fef4a1c02eb4e70b4d4ea1f03310385, entries=150, sequenceid=70, filesize=11.6 K 2023-04-20 20:55:24,899 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/1773933967db4efcbe8f3bf90db89a8b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1773933967db4efcbe8f3bf90db89a8b 2023-04-20 20:55:24,910 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1773933967db4efcbe8f3bf90db89a8b, entries=150, sequenceid=70, filesize=11.6 K 2023-04-20 20:55:24,911 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 90ms, sequenceid=70, compaction requested=true 2023-04-20 20:55:24,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:24,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:24,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:24,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:24,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,914 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:24,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5919feb[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:24,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:24,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:24,919 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,919 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:24,921 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:24,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:24,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:24,923 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:24,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:24,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:24,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:24,924 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:24,926 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,926 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,926 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:24,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:24,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:24,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:24,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:24,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:24,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:24,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:24,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:24,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:24,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:24,932 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:24,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:24,932 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:24,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:24,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,032 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:25,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:25,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:25,036 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:25,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:25,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:25,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,039 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,039 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,039 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:25,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:25,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3520209b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:25,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:25,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:25,047 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:25,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:25,048 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:25,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:25,051 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:25,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:25,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:25,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:25,051 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:25,053 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,053 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,054 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:25,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:25,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:25,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,059 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,059 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:25,159 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:25,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:25,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:25,163 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:25,163 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,163 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:25,163 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:25,164 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,164 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,165 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:25,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:25,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:25,197 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,197 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:25,197 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,197 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:25,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7151910e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:25,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:25,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:25,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:25,208 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:25,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:25,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:25,212 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:25,212 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:25,212 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:25,212 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:25,212 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:25,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,217 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/07b9bd9d07dc4afd9c3bb722db0dc644 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/07b9bd9d07dc4afd9c3bb722db0dc644 2023-04-20 20:55:25,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,222 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 07b9bd9d07dc4afd9c3bb722db0dc644(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:25,222 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,222 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=13, startTime=1682024124796; duration=0sec 2023-04-20 20:55:25,222 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:25,222 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,226 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:25,226 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:25,226 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,226 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/b52e91fec7154df6aec60019bd0532c2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/05f2dfb8a1c44eb293ae7e9230c9f9aa, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a085f0ff2a084ea4a15898f0903f4f51, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1773933967db4efcbe8f3bf90db89a8b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=46.4 K 2023-04-20 20:55:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,228 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b52e91fec7154df6aec60019bd0532c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682024121257 2023-04-20 20:55:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,229 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 05f2dfb8a1c44eb293ae7e9230c9f9aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682024121277 2023-04-20 20:55:25,230 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a085f0ff2a084ea4a15898f0903f4f51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682024123417 2023-04-20 20:55:25,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,231 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1773933967db4efcbe8f3bf90db89a8b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682024123542 2023-04-20 20:55:25,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,231 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:25,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:25,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:25,232 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:25,232 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:25,240 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#304 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:25,284 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/36f9b307168c4255a98c06f6ab8334e7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/36f9b307168c4255a98c06f6ab8334e7 2023-04-20 20:55:25,284 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/9d0beb4c7725437ab88d2c7e07903024 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9d0beb4c7725437ab88d2c7e07903024 2023-04-20 20:55:25,289 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 36f9b307168c4255a98c06f6ab8334e7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:25,289 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,289 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=12, startTime=1682024124796; duration=0sec 2023-04-20 20:55:25,290 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:25,290 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 9d0beb4c7725437ab88d2c7e07903024(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:25,290 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,290 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=13, startTime=1682024124796; duration=0sec 2023-04-20 20:55:25,290 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:25,332 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:25,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:25,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:25,336 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:25,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:25,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:25,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,340 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,340 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:25,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:25,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:25,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,344 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:25,346 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee8cd9a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:25,348 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:25,348 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,350 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:25,350 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,350 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:25,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:25,355 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:25,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:25,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:25,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:25,355 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:25,357 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,364 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:25,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:25,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:25,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,365 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,365 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:25,365 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,365 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:25,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,465 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:25,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:25,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:25,469 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:25,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:25,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:25,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,470 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,471 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:25,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:25,473 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,473 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:25,473 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,473 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,475 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@678f796b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:25,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:25,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:25,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:25,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:25,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:25,485 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:25,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:25,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:25,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:25,485 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:25,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:25,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:25,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,492 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:25,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,492 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,492 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:25,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:25,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,593 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:25,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:25,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:25,596 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:25,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:25,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:25,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,598 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,598 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,599 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:25,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:25,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:25,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:25,601 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,601 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:25,602 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:25,602 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:25,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:25,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@539fb9f9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:25,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:25,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,607 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:25,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:25,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:25,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:25,610 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:25,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:25,613 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:25,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:25,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:25,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:25,613 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:25,615 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,615 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,615 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:25,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,615 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:25,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:25,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:25,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:25,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:25,628 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:25,628 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:25,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:25,728 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:25,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:25,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:25,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:25,732 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:25,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:25,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:25,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:25,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:25,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:25,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:25,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:25,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:25,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:25,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:25,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:25,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:25,736 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-20 20:55:25,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:25,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:25,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:25,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:25,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:25,737 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:25,753 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d3775a7cf855449496d85847ef9ed1b1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d3775a7cf855449496d85847ef9ed1b1 2023-04-20 20:55:25,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:25,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:25,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024125784, max timestamp=1682024125798Num uniques 859; 2023-04-20 20:55:25,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:25,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024125784, max timestamp=1682024125798Num uniques 859; 2023-04-20 20:55:25,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:25,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024125784, max timestamp=1682024125798Num uniques 859; 2023-04-20 20:55:25,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:25,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024185827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:25,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024185832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024125825, max timestamp=1682024125836Num uniques 1068; 2023-04-20 20:55:25,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:55:25,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:25,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024125825, max timestamp=1682024125836Num uniques 1068; 2023-04-20 20:55:25,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024185838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:55:25,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024125825, max timestamp=1682024125836Num uniques 1068; 2023-04-20 20:55:25,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-20 20:55:25,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024125858, max timestamp=1682024125879Num uniques 1027; 2023-04-20 20:55:25,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-20 20:55:25,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024125858, max timestamp=1682024125879Num uniques 1027; 2023-04-20 20:55:25,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024125858, max timestamp=1682024125879Num uniques 1009; 2023-04-20 20:55:25,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-20 20:55:25,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10392, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-04-20 20:55:25,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:25,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024185929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:25,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024125915, max timestamp=1682024125933Num uniques 1086; 2023-04-20 20:55:25,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:55:25,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024125915, max timestamp=1682024125933Num uniques 1086; 2023-04-20 20:55:25,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682024125918, max timestamp=1682024125933Num uniques 1086; 2023-04-20 20:55:25,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:55:25,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-20 20:55:25,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682024125957, max timestamp=1682024125967Num uniques 1095; 2023-04-20 20:55:25,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682024125957, max timestamp=1682024125967Num uniques 1045; 2023-04-20 20:55:25,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-20 20:55:25,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-20 20:55:25,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682024125957, max timestamp=1682024125967Num uniques 1095; 2023-04-20 20:55:25,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-20 20:55:26,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024186001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024186002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024186003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024186002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024186035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024186115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024186115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024186115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024186115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,165 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/c6975b878a4f401680dcde751f3e9f17 2023-04-20 20:55:26,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024186241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024186323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024186324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024186324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024186324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024186549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,589 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/7a9a63b0dcd44ecd979169b3a538a8b7 2023-04-20 20:55:26,598 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/01dc8d74916b4d63a1fdedd23456c8b2 2023-04-20 20:55:26,603 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/c6975b878a4f401680dcde751f3e9f17 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/c6975b878a4f401680dcde751f3e9f17 2023-04-20 20:55:26,611 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/c6975b878a4f401680dcde751f3e9f17, entries=250, sequenceid=86, filesize=38.7 K 2023-04-20 20:55:26,611 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/7a9a63b0dcd44ecd979169b3a538a8b7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a9a63b0dcd44ecd979169b3a538a8b7 2023-04-20 20:55:26,616 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a9a63b0dcd44ecd979169b3a538a8b7, entries=100, sequenceid=86, filesize=9.3 K 2023-04-20 20:55:26,617 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/01dc8d74916b4d63a1fdedd23456c8b2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/01dc8d74916b4d63a1fdedd23456c8b2 2023-04-20 20:55:26,621 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/01dc8d74916b4d63a1fdedd23456c8b2, entries=100, sequenceid=86, filesize=9.3 K 2023-04-20 20:55:26,621 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=161.02 KB/164880 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 885ms, sequenceid=86, compaction requested=true 2023-04-20 20:55:26,621 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:26,622 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:26,625 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,625 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:26,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:26,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ebb9b65[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:26,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:26,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,629 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:26,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:26,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:26,632 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:26,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:26,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:26,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:26,632 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:26,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024125993, max timestamp=1682024126632Num uniques 1204; 2023-04-20 20:55:26,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-20 20:55:26,635 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,635 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:26,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,635 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:26,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:26,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024125993, max timestamp=1682024126632Num uniques 1204; 2023-04-20 20:55:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-20 20:55:26,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:26,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=73.80 KB heapSize=200.42 KB 2023-04-20 20:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,637 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024125993, max timestamp=1682024126632Num uniques 1204; 2023-04-20 20:55:26,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-20 20:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:26,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:26,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:26,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:26,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:26,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:26,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:26,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:26,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:26,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:26,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:26,642 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:26,642 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:26,642 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:26,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:26,676 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420fe55254002b2465392d8f636df5b2fc3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fe55254002b2465392d8f636df5b2fc3 2023-04-20 20:55:26,692 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/5cee5c2446f14ef78de8b3f09b829fe9 2023-04-20 20:55:26,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/266d02320bde4fa28bd6c78d09da0137 2023-04-20 20:55:26,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024126682, max timestamp=1682024126719Num uniques 859; 2023-04-20 20:55:26,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:26,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024126682, max timestamp=1682024126718Num uniques 859; 2023-04-20 20:55:26,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:26,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024126682, max timestamp=1682024126718Num uniques 859; 2023-04-20 20:55:26,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:26,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/f7d99880114549e5a8ef732e3e638f76 2023-04-20 20:55:26,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/5cee5c2446f14ef78de8b3f09b829fe9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5cee5c2446f14ef78de8b3f09b829fe9 2023-04-20 20:55:26,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5cee5c2446f14ef78de8b3f09b829fe9, entries=200, sequenceid=195, filesize=32.1 K 2023-04-20 20:55:26,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/266d02320bde4fa28bd6c78d09da0137 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/266d02320bde4fa28bd6c78d09da0137 2023-04-20 20:55:26,742 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:26,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:26,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:26,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:26,746 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:26,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:26,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:26,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:26,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/266d02320bde4fa28bd6c78d09da0137, entries=150, sequenceid=195, filesize=11.8 K 2023-04-20 20:55:26,747 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:26,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/f7d99880114549e5a8ef732e3e638f76 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f7d99880114549e5a8ef732e3e638f76 2023-04-20 20:55:26,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:26,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f7d99880114549e5a8ef732e3e638f76, entries=150, sequenceid=195, filesize=11.8 K 2023-04-20 20:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.50 KB/109056, currentSize=127.47 KB/130530 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 118ms, sequenceid=195, compaction requested=true 2023-04-20 20:55:26,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:26,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:26,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:26,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:26,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 20:55:26,755 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-20 20:55:26,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:26,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:26,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:26,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:26,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:26,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:26,755 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:26,754 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6970735[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:26,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:26,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,757 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45453 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:26,757 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:26,757 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,757 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/07b9bd9d07dc4afd9c3bb722db0dc644, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/9fef4a1c02eb4e70b4d4ea1f03310385, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a9a63b0dcd44ecd979169b3a538a8b7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/266d02320bde4fa28bd6c78d09da0137] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=44.4 K 2023-04-20 20:55:26,758 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 124107 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:26,758 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:26,758 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 07b9bd9d07dc4afd9c3bb722db0dc644, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682024123417 2023-04-20 20:55:26,758 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,758 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9d0beb4c7725437ab88d2c7e07903024, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/4b5c34f679d745d8b91257e8df539db9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/c6975b878a4f401680dcde751f3e9f17, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5cee5c2446f14ef78de8b3f09b829fe9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=121.2 K 2023-04-20 20:55:26,758 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,758 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:26,758 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:26,758 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:26,758 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9fef4a1c02eb4e70b4d4ea1f03310385, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682024123542 2023-04-20 20:55:26,758 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9d0beb4c7725437ab88d2c7e07903024, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1682024123417 2023-04-20 20:55:26,759 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7a9a63b0dcd44ecd979169b3a538a8b7, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682024125722 2023-04-20 20:55:26,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,759 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 4b5c34f679d745d8b91257e8df539db9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682024123542 2023-04-20 20:55:26,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:26,759 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 266d02320bde4fa28bd6c78d09da0137, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682024126001 2023-04-20 20:55:26,759 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting c6975b878a4f401680dcde751f3e9f17, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682024125719 2023-04-20 20:55:26,760 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5cee5c2446f14ef78de8b3f09b829fe9, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682024126001 2023-04-20 20:55:26,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:26,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:26,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:26,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:26,764 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:26,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:26,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:26,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:26,764 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:26,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:26,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:26,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:26,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:26,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:26,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:26,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,771 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:26,771 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:26,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:26,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:26,815 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#312 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:26,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024186770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024186772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024186809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024186816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,833 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/91184d7b3a584cc291e848befefd7720 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/91184d7b3a584cc291e848befefd7720 2023-04-20 20:55:26,834 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304200b2b65d4e357471986d7cb19c8403ae7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200b2b65d4e357471986d7cb19c8403ae7 2023-04-20 20:55:26,838 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 91184d7b3a584cc291e848befefd7720(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:26,838 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:26,838 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=12, startTime=1682024126754; duration=0sec 2023-04-20 20:55:26,838 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:26,839 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:26,840 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#313 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:26,844 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33602 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:26,845 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:26,845 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,845 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/36f9b307168c4255a98c06f6ab8334e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/01dc8d74916b4d63a1fdedd23456c8b2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f7d99880114549e5a8ef732e3e638f76] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=32.8 K 2023-04-20 20:55:26,845 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 36f9b307168c4255a98c06f6ab8334e7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682024123542 2023-04-20 20:55:26,846 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 01dc8d74916b4d63a1fdedd23456c8b2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682024125722 2023-04-20 20:55:26,846 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f7d99880114549e5a8ef732e3e638f76, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682024126001 2023-04-20 20:55:26,861 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#314 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:26,862 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/da6c5a06357944d98843e9b9a0800534 2023-04-20 20:55:26,872 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:26,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:26,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:26,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:26,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:26,877 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:26,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:26,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:26,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:26,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,879 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:26,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:26,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:26,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:26,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:26,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:26,886 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:26,886 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:26,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/d3f1e051df484ae0871f3874d97a14e0 2023-04-20 20:55:26,888 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,889 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:26,889 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:26,889 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:26,891 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,891 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:26,892 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,892 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a4f72ad[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:26,894 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:26,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,897 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:26,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:26,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:26,899 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:26,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:26,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:26,899 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:26,899 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:26,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:26,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:26,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:26,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,904 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:26,906 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:26,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:26,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:26,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:26,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:26,906 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:26,906 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:26,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024186917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024186917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024186918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:26,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:26,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024186926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,007 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:27,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:27,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:27,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:27,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:27,010 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:27,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:27,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:27,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,011 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,012 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:27,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:27,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:27,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:27,024 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,024 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:27,024 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:27,024 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:27,026 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3431506c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:27,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:27,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:27,030 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,030 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:27,038 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,040 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:27,044 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:27,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:27,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:27,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:27,044 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:27,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:27,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:27,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:27,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:27,052 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:27,052 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:27,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024187053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024187124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024187125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024187126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024187137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,152 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:27,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:27,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:27,156 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:27,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:27,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:27,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,158 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:27,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:27,160 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:27,162 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:27,164 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a3853c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:27,164 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,165 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:27,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:27,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:27,167 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:27,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,167 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,167 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:27,170 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:27,171 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:27,171 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:27,171 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:27,171 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:27,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,172 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,173 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:27,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,178 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:27,178 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:27,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:27,274 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/0526f15c1f4445e19dee2b44031f82bf as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0526f15c1f4445e19dee2b44031f82bf 2023-04-20 20:55:27,279 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:27,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:27,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:27,281 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 0526f15c1f4445e19dee2b44031f82bf(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:27,281 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:27,281 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=13, startTime=1682024126754; duration=0sec 2023-04-20 20:55:27,282 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:27,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:27,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:27,282 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,284 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:27,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:27,285 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,285 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:27,286 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:27,286 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:27,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78339c72[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:27,289 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:27,289 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,290 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:27,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:27,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:27,296 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:27,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:27,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:27,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:27,297 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:27,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:27,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:27,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,303 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:27,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,303 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:27,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:27,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,318 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/e6ba9d75e11e42958f3a8a3c99a10b3c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/e6ba9d75e11e42958f3a8a3c99a10b3c 2023-04-20 20:55:27,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/69f2124ebca249a79108ce4297bf2f2e 2023-04-20 20:55:27,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/da6c5a06357944d98843e9b9a0800534 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/da6c5a06357944d98843e9b9a0800534 2023-04-20 20:55:27,327 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into e6ba9d75e11e42958f3a8a3c99a10b3c(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:27,327 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:27,327 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=12, startTime=1682024126754; duration=0sec 2023-04-20 20:55:27,327 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:27,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/da6c5a06357944d98843e9b9a0800534, entries=300, sequenceid=230, filesize=45.8 K 2023-04-20 20:55:27,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/d3f1e051df484ae0871f3874d97a14e0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/d3f1e051df484ae0871f3874d97a14e0 2023-04-20 20:55:27,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/d3f1e051df484ae0871f3874d97a14e0, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 20:55:27,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/69f2124ebca249a79108ce4297bf2f2e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/69f2124ebca249a79108ce4297bf2f2e 2023-04-20 20:55:27,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/69f2124ebca249a79108ce4297bf2f2e, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 20:55:27,344 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 590ms, sequenceid=230, compaction requested=false 2023-04-20 20:55:27,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:27,403 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:27,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:27,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:27,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:27,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:27,407 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:27,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:27,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:27,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,409 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:27,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:27,412 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:27,412 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:27,412 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:27,412 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:27,431 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304207de3eb84cb8c40bd8a4ef5ef1f1d609d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207de3eb84cb8c40bd8a4ef5ef1f1d609d 2023-04-20 20:55:27,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:27,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:27,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024127459, max timestamp=1682024127483Num uniques 859; 2023-04-20 20:55:27,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:55:27,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024127459, max timestamp=1682024127483Num uniques 859; 2023-04-20 20:55:27,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024127459, max timestamp=1682024127483Num uniques 859; 2023-04-20 20:55:27,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:55:27,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:55:27,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024187505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024187511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024187512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024187512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024187613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024187614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024187621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024187621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024187818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024187819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024187826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:27,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024187826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,842 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/dec4f0d3bd60467d8c6a4d412631bc7b 2023-04-20 20:55:27,861 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/a1662b221cf643f8aab1413e27e084e0 2023-04-20 20:55:27,892 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/4300a471364c4990a5d629fe2ead5bc8 2023-04-20 20:55:27,895 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/dec4f0d3bd60467d8c6a4d412631bc7b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dec4f0d3bd60467d8c6a4d412631bc7b 2023-04-20 20:55:27,899 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dec4f0d3bd60467d8c6a4d412631bc7b, entries=100, sequenceid=243, filesize=18.5 K 2023-04-20 20:55:27,899 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/a1662b221cf643f8aab1413e27e084e0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a1662b221cf643f8aab1413e27e084e0 2023-04-20 20:55:27,904 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a1662b221cf643f8aab1413e27e084e0, entries=100, sequenceid=243, filesize=9.4 K 2023-04-20 20:55:27,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/4300a471364c4990a5d629fe2ead5bc8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/4300a471364c4990a5d629fe2ead5bc8 2023-04-20 20:55:27,911 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/4300a471364c4990a5d629fe2ead5bc8, entries=100, sequenceid=243, filesize=9.4 K 2023-04-20 20:55:27,911 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 499ms, sequenceid=243, compaction requested=true 2023-04-20 20:55:27,912 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:27,912 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:27,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@198843c3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:27,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:27,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,917 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:27,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:27,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:27,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:27,922 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:27,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:27,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:27,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:27,922 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:27,923 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,923 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:27,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,923 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:27,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,925 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:27,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:27,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:27,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:27,932 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:27,932 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:27,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:28,032 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:28,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:28,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:28,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:28,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:28,036 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:28,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:28,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:28,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:28,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:28,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,041 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:28,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:28,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:28,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:28,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:28,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:28,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:28,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:28,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:28,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:28,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:28,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:28,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:28,043 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-20 20:55:28,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:28,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:28,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:28,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:28,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:28,044 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:28,065 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420e465f15424c0457db4eaf917188c6bb6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e465f15424c0457db4eaf917188c6bb6 2023-04-20 20:55:28,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:28,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:28,074 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/98df15fbc504450cae97cdd6221b02c1 2023-04-20 20:55:28,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024188111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024188122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024188123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024188133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024188134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024188226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024188432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,490 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/1e78124cd638450cb11a1b3ecdc82379 2023-04-20 20:55:28,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024188628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024188629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024188641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024188642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:28,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44600 deadline: 1682024188739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,901 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e53dbd24349c489692ac20f2e272799b 2023-04-20 20:55:28,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/98df15fbc504450cae97cdd6221b02c1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/98df15fbc504450cae97cdd6221b02c1 2023-04-20 20:55:28,909 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/98df15fbc504450cae97cdd6221b02c1, entries=150, sequenceid=279, filesize=25.4 K 2023-04-20 20:55:28,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/1e78124cd638450cb11a1b3ecdc82379 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/1e78124cd638450cb11a1b3ecdc82379 2023-04-20 20:55:28,914 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/1e78124cd638450cb11a1b3ecdc82379, entries=150, sequenceid=279, filesize=11.9 K 2023-04-20 20:55:28,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e53dbd24349c489692ac20f2e272799b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e53dbd24349c489692ac20f2e272799b 2023-04-20 20:55:28,918 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e53dbd24349c489692ac20f2e272799b, entries=150, sequenceid=279, filesize=11.9 K 2023-04-20 20:55:28,919 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 876ms, sequenceid=279, compaction requested=true 2023-04-20 20:55:28,919 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:28,919 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:28,920 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,920 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:28,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:28,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:28,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2784d2ae[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:28,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:28,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:28,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:28,926 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:28,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:28,927 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:28,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:28,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:28,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:28,927 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:28,929 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,929 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,929 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:28,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:28,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:28,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:28,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:28,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:28,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:28,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:28,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:28,938 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:28,939 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:28,939 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:28,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:29,039 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:29,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:29,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:29,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:29,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:29,048 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:29,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:29,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:29,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:29,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:29,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:29,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:29,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:29,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:29,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:29,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:29,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:29,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:29,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:29,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:29,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:29,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:29,056 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-20 20:55:29,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:29,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:29,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:29,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:29,057 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:29,057 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:29,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:29,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024129355, max timestamp=1682024129382Num uniques 859; 2023-04-20 20:55:29,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:29,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024129355, max timestamp=1682024129382Num uniques 859; 2023-04-20 20:55:29,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:29,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024129355, max timestamp=1682024129382Num uniques 859; 2023-04-20 20:55:29,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:29,471 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420e963a736811e4cb697cf74ff80de4eb4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e963a736811e4cb697cf74ff80de4eb4 2023-04-20 20:55:29,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024129475, max timestamp=1682024129501Num uniques 1018; 2023-04-20 20:55:29,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 20:55:29,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024129475, max timestamp=1682024129501Num uniques 1018; 2023-04-20 20:55:29,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024129475, max timestamp=1682024129501Num uniques 1018; 2023-04-20 20:55:29,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 20:55:29,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 20:55:29,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024129572, max timestamp=1682024129637Num uniques 1027; 2023-04-20 20:55:29,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024129572, max timestamp=1682024129637Num uniques 1027; 2023-04-20 20:55:29,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:55:29,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:55:29,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024129572, max timestamp=1682024129637Num uniques 1027; 2023-04-20 20:55:29,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:55:29,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024189666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024189666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1682024129656, max timestamp=1682024129669Num uniques 1036; 2023-04-20 20:55:29,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-20 20:55:29,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1682024129656, max timestamp=1682024129669Num uniques 1036; 2023-04-20 20:55:29,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-20 20:55:29,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1682024129656, max timestamp=1682024129669Num uniques 1036; 2023-04-20 20:55:29,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-20 20:55:29,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024189706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024189714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024189715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024189772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024189773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024189816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024189818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024189823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,888 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=288, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/fc27ea171f4e4b53802950f21d0ec1d0 2023-04-20 20:55:29,917 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/52685ee295134f66a83019c11e827ae0 2023-04-20 20:55:29,928 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/68cc14a4e38f4a83af2f641c119869b6 2023-04-20 20:55:29,934 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/fc27ea171f4e4b53802950f21d0ec1d0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/fc27ea171f4e4b53802950f21d0ec1d0 2023-04-20 20:55:29,937 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/fc27ea171f4e4b53802950f21d0ec1d0, entries=150, sequenceid=288, filesize=25.4 K 2023-04-20 20:55:29,938 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/52685ee295134f66a83019c11e827ae0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/52685ee295134f66a83019c11e827ae0 2023-04-20 20:55:29,941 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/52685ee295134f66a83019c11e827ae0, entries=150, sequenceid=288, filesize=11.9 K 2023-04-20 20:55:29,942 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/68cc14a4e38f4a83af2f641c119869b6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/68cc14a4e38f4a83af2f641c119869b6 2023-04-20 20:55:29,946 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/68cc14a4e38f4a83af2f641c119869b6, entries=150, sequenceid=288, filesize=11.9 K 2023-04-20 20:55:29,947 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 891ms, sequenceid=288, compaction requested=true 2023-04-20 20:55:29,947 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:29,947 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:29,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:29,950 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:29,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:29,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dd43907[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:29,951 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:29,951 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,952 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:29,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:29,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:29,954 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:29,955 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:29,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:29,957 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:29,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:29,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:29,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:29,957 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:29,958 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,958 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:29,959 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:29,959 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,960 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:29,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:29,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:29,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:29,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:29,965 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:29,965 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:29,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:29,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:29,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:29,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=154.31 KB heapSize=423.47 KB 2023-04-20 20:55:29,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:29,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:29,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:29,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:29,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:29,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:30,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024190006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024190007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024190024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024190026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024190032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,065 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:30,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:30,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:30,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:30,069 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:30,069 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,069 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:30,069 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:30,069 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,069 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:30,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:30,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:30,073 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,073 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:30,073 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:30,073 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:30,074 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,074 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:30,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1960fac3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:30,076 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:30,076 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,077 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:30,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:30,087 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:30,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:30,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:30,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:30,087 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:30,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,090 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:30,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:30,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:30,096 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:30,096 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:30,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024190115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024190115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,197 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:30,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:30,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:30,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:30,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:30,201 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:30,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:30,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:30,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,204 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:30,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:30,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:30,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:30,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:30,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:30,222 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,222 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:30,222 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:30,222 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:30,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:30,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:30,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:30,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:30,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@422d2477[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:30,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:30,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:30,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:30,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:30,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:30,232 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:30,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:30,237 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:30,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:30,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:30,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:30,237 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:30,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,249 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:30,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:30,249 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,249 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,249 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:30,249 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:30,249 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:30,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44534 deadline: 1682024190326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024190326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024190334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024190335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,349 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:30,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:30,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:30,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024190342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:30,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:30,359 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:30,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:30,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:30,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:30,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:30,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:30,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:30,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:30,376 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,377 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,377 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:30,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@433d9f3e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:30,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:30,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:30,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:30,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:30,387 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:30,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:30,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:30,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:30,387 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:30,390 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,391 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,391 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,391 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,392 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,395 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:30,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:30,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,395 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:30,396 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:30,396 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,396 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,404 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420ed69c7de08d9463e80f55d347daa8e68 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ed69c7de08d9463e80f55d347daa8e68 2023-04-20 20:55:30,420 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=375, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/0bcd44da4904447cb5e5ab3e1f6d93c5 2023-04-20 20:55:30,496 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:30,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:30,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:30,501 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:30,501 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,501 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:30,501 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:30,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,507 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,507 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:30,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:30,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:30,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:30,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:30,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:30,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:30,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:30,518 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,518 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:30,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7d8a5f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:30,522 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:30,522 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,524 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:30,525 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,525 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:30,528 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:30,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:30,531 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:30,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:30,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:30,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:30,531 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:30,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,534 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,536 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,540 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:30,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:30,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:30,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,540 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:30,540 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:30,640 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:30,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:30,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024190635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44530 deadline: 1682024190637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:30,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:30,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:30,648 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:30,648 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,648 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:30,648 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:30,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,654 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,654 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:30,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:30,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,656 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:30,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:30,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:30,660 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,660 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:30,660 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:30,660 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:30,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:30,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:30,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:30,663 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,663 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:30,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155dd91[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:30,665 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:30,665 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,667 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,667 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:30,667 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:30,667 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:30,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,668 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:30,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:30,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:30,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:30,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:30,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:30,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:30,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:30,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:30,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,682 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:30,682 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:30,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,783 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:30,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:30,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:30,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:30,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:30,788 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:30,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:30,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:30,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:30,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:30,791 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,791 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:30,791 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:30,792 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:30,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:30,793 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,793 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,793 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:30,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a705f50[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:30,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:30,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,796 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,796 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:30,796 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:30,797 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:30,797 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:30,799 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,799 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:30,799 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,799 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:30,801 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:30,801 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:30,801 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:30,802 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:30,802 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:30,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:30,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:30,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,805 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,809 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:30,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:30,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:30,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,809 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:30,809 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:30,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:30,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:30,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024190845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/a158895da6d94024a86e2032615b397a 2023-04-20 20:55:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024190849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:30,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024190855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/f107ea01716b4e65ae7dedd00d6d295e 2023-04-20 20:55:30,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/0bcd44da4904447cb5e5ab3e1f6d93c5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0bcd44da4904447cb5e5ab3e1f6d93c5 2023-04-20 20:55:30,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0bcd44da4904447cb5e5ab3e1f6d93c5, entries=200, sequenceid=375, filesize=32.3 K 2023-04-20 20:55:30,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/a158895da6d94024a86e2032615b397a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a158895da6d94024a86e2032615b397a 2023-04-20 20:55:30,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a158895da6d94024a86e2032615b397a, entries=150, sequenceid=375, filesize=11.9 K 2023-04-20 20:55:30,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/f107ea01716b4e65ae7dedd00d6d295e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f107ea01716b4e65ae7dedd00d6d295e 2023-04-20 20:55:30,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f107ea01716b4e65ae7dedd00d6d295e, entries=150, sequenceid=375, filesize=11.9 K 2023-04-20 20:55:30,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~423.42 KB/433584, currentSize=33.54 KB/34350 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 899ms, sequenceid=375, compaction requested=true 2023-04-20 20:55:30,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:30,889 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:30,889 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:30,889 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:30,889 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:30,889 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:30,891 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 177082 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:30,891 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:30,891 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,891 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/e6ba9d75e11e42958f3a8a3c99a10b3c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/da6c5a06357944d98843e9b9a0800534, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dec4f0d3bd60467d8c6a4d412631bc7b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/98df15fbc504450cae97cdd6221b02c1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/fc27ea171f4e4b53802950f21d0ec1d0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0bcd44da4904447cb5e5ab3e1f6d93c5] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=172.9 K 2023-04-20 20:55:30,892 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70512 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:30,892 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:30,892 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,892 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/91184d7b3a584cc291e848befefd7720, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/d3f1e051df484ae0871f3874d97a14e0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a1662b221cf643f8aab1413e27e084e0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/1e78124cd638450cb11a1b3ecdc82379, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/52685ee295134f66a83019c11e827ae0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a158895da6d94024a86e2032615b397a] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=68.9 K 2023-04-20 20:55:30,892 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e6ba9d75e11e42958f3a8a3c99a10b3c, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682024126001 2023-04-20 20:55:30,893 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 91184d7b3a584cc291e848befefd7720, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682024126001 2023-04-20 20:55:30,893 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting da6c5a06357944d98843e9b9a0800534, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682024126736 2023-04-20 20:55:30,893 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d3f1e051df484ae0871f3874d97a14e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682024126736 2023-04-20 20:55:30,893 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting dec4f0d3bd60467d8c6a4d412631bc7b, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682024126768 2023-04-20 20:55:30,894 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a1662b221cf643f8aab1413e27e084e0, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682024126768 2023-04-20 20:55:30,894 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1e78124cd638450cb11a1b3ecdc82379, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682024127510 2023-04-20 20:55:30,894 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 52685ee295134f66a83019c11e827ae0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682024128072 2023-04-20 20:55:30,895 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 98df15fbc504450cae97cdd6221b02c1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682024127510 2023-04-20 20:55:30,895 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a158895da6d94024a86e2032615b397a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682024129705 2023-04-20 20:55:30,895 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting fc27ea171f4e4b53802950f21d0ec1d0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682024128072 2023-04-20 20:55:30,896 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0bcd44da4904447cb5e5ab3e1f6d93c5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682024129705 2023-04-20 20:55:30,910 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:30,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:30,914 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#329 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:30,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:30,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:30,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:30,920 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:30,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:30,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:30,921 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,921 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,922 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:30,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:30,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:30,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:30,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:30,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:30,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:30,924 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:30,924 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:30,925 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-20 20:55:30,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:30,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:30,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:30,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:30,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:30,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:30,928 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#330 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:30,949 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420730deb53b2dd4a3293f2d0a72d9814e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420730deb53b2dd4a3293f2d0a72d9814e7 2023-04-20 20:55:31,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:31,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:31,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024131245, max timestamp=1682024131264Num uniques 859; 2023-04-20 20:55:31,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:31,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024131245, max timestamp=1682024131264Num uniques 859; 2023-04-20 20:55:31,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:31,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024131245, max timestamp=1682024131264Num uniques 859; 2023-04-20 20:55:31,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:31,326 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/0405c64a28d740f6bfe593698628672b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0405c64a28d740f6bfe593698628672b 2023-04-20 20:55:31,330 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 0405c64a28d740f6bfe593698628672b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:31,330 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:31,330 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=10, startTime=1682024130889; duration=0sec 2023-04-20 20:55:31,330 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:31,330 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:55:31,332 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70512 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:55:31,332 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:31,332 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:31,332 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0526f15c1f4445e19dee2b44031f82bf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/69f2124ebca249a79108ce4297bf2f2e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/4300a471364c4990a5d629fe2ead5bc8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e53dbd24349c489692ac20f2e272799b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/68cc14a4e38f4a83af2f641c119869b6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f107ea01716b4e65ae7dedd00d6d295e] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=68.9 K 2023-04-20 20:55:31,332 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0526f15c1f4445e19dee2b44031f82bf, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682024126001 2023-04-20 20:55:31,333 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 69f2124ebca249a79108ce4297bf2f2e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682024126736 2023-04-20 20:55:31,333 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4300a471364c4990a5d629fe2ead5bc8, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682024126768 2023-04-20 20:55:31,333 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e53dbd24349c489692ac20f2e272799b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682024127510 2023-04-20 20:55:31,334 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 68cc14a4e38f4a83af2f641c119869b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1682024128072 2023-04-20 20:55:31,334 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f107ea01716b4e65ae7dedd00d6d295e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682024129705 2023-04-20 20:55:31,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024131316, max timestamp=1682024131338Num uniques 1068; 2023-04-20 20:55:31,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:55:31,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024131316, max timestamp=1682024131338Num uniques 1068; 2023-04-20 20:55:31,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:55:31,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682024131316, max timestamp=1682024131338Num uniques 1068; 2023-04-20 20:55:31,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 20:55:31,346 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#332 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:31,354 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=383, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/862e2f606fb049ea9d6803ae567af72b 2023-04-20 20:55:31,357 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/622acff34b984d3a92a17a93a89ac0d1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/622acff34b984d3a92a17a93a89ac0d1 2023-04-20 20:55:31,363 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 622acff34b984d3a92a17a93a89ac0d1(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:31,363 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:31,363 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=10, startTime=1682024130889; duration=0sec 2023-04-20 20:55:31,363 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:31,377 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/2f6912366c8e472f9e4cc51595bb4125 2023-04-20 20:55:31,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024131392, max timestamp=1682024131424Num uniques 1027; 2023-04-20 20:55:31,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:55:31,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024131392, max timestamp=1682024131424Num uniques 1027; 2023-04-20 20:55:31,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:55:31,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024131392, max timestamp=1682024131424Num uniques 1027; 2023-04-20 20:55:31,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-20 20:55:31,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024131473, max timestamp=1682024131492Num uniques 1036; 2023-04-20 20:55:31,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-20 20:55:31,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024131473, max timestamp=1682024131492Num uniques 1036; 2023-04-20 20:55:31,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-20 20:55:31,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024131473, max timestamp=1682024131492Num uniques 1036; 2023-04-20 20:55:31,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-20 20:55:31,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024131533, max timestamp=1682024131547Num uniques 1095; 2023-04-20 20:55:31,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024131533, max timestamp=1682024131547Num uniques 1095; 2023-04-20 20:55:31,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 20:55:31,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682024131533, max timestamp=1682024131547Num uniques 1095; 2023-04-20 20:55:31,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 20:55:31,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 20:55:31,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024131587, max timestamp=1682024131606Num uniques 1054; 2023-04-20 20:55:31,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:55:31,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024191602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024131587, max timestamp=1682024131606Num uniques 1054; 2023-04-20 20:55:31,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682024131587, max timestamp=1682024131606Num uniques 1054; 2023-04-20 20:55:31,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:55:31,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-20 20:55:31,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682024131687, max timestamp=1682024131718Num uniques 1013; 2023-04-20 20:55:31,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-20 20:55:31,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682024131687, max timestamp=1682024131718Num uniques 1013; 2023-04-20 20:55:31,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-20 20:55:31,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682024131687, max timestamp=1682024131718Num uniques 1013; 2023-04-20 20:55:31,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-20 20:55:31,755 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/5c67c05ef3444465b2b9d835b6a16218 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/5c67c05ef3444465b2b9d835b6a16218 2023-04-20 20:55:31,759 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 5c67c05ef3444465b2b9d835b6a16218(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:31,759 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:31,759 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=10, startTime=1682024130889; duration=0sec 2023-04-20 20:55:31,759 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:31,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682024131765, max timestamp=1682024131782Num uniques 1022; 2023-04-20 20:55:31,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682024131765, max timestamp=1682024131782Num uniques 1022; 2023-04-20 20:55:31,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-20 20:55:31,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-20 20:55:31,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682024131765, max timestamp=1682024131782Num uniques 1022; 2023-04-20 20:55:31,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-20 20:55:31,813 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/113cdf5d3a014f36afa9a7b4905fd796 2023-04-20 20:55:31,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/862e2f606fb049ea9d6803ae567af72b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/862e2f606fb049ea9d6803ae567af72b 2023-04-20 20:55:31,821 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/862e2f606fb049ea9d6803ae567af72b, entries=100, sequenceid=383, filesize=18.6 K 2023-04-20 20:55:31,821 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/2f6912366c8e472f9e4cc51595bb4125 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2f6912366c8e472f9e4cc51595bb4125 2023-04-20 20:55:31,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024131795, max timestamp=1682024131824Num uniques 1081; 2023-04-20 20:55:31,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-20 20:55:31,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024131795, max timestamp=1682024131824Num uniques 1081; 2023-04-20 20:55:31,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-20 20:55:31,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024131795, max timestamp=1682024131824Num uniques 1081; 2023-04-20 20:55:31,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-20 20:55:31,827 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2f6912366c8e472f9e4cc51595bb4125, entries=100, sequenceid=383, filesize=9.5 K 2023-04-20 20:55:31,828 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/113cdf5d3a014f36afa9a7b4905fd796 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/113cdf5d3a014f36afa9a7b4905fd796 2023-04-20 20:55:31,839 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/113cdf5d3a014f36afa9a7b4905fd796, entries=100, sequenceid=383, filesize=9.5 K 2023-04-20 20:55:31,839 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=60.38 KB/61830 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 914ms, sequenceid=383, compaction requested=false 2023-04-20 20:55:31,839 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:31,839 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:31,839 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:31,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:31,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=67.09 KB heapSize=179.02 KB 2023-04-20 20:55:31,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:31,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:31,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:31,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:31,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:31,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:31,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:31,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:31,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:31,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51be14f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:31,845 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:31,845 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,852 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:31,852 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,852 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:31,854 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:31,858 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304206a59f3d71a304a129e0cf824c96dbcbd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206a59f3d71a304a129e0cf824c96dbcbd 2023-04-20 20:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:31,866 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:31,866 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:31,866 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:31,866 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:31,866 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:31,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:31,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:31,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:31,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,878 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,878 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:31,878 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:31,878 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:31,878 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:31,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:31,878 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,879 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,878 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:31,879 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:31,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:31,879 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:31,879 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:31,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:31,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024191873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024191875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,887 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=550, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/766bfe21275f43f4b4cc0ed8c7b2df65 2023-04-20 20:55:31,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024191881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024191884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024191886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/be789d1bea9846449c6eb0faf8ae85db 2023-04-20 20:55:31,979 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:31,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:31,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:31,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:31,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:31,984 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:31,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:31,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:31,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:31,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,985 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:31,987 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:31,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:31,987 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:31,987 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:31,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:31,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:31,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:31,988 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:31,989 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:31,989 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:31,989 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:31,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:31,993 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:31,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,993 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:31,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024191987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:31,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:31,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024191987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c7bcc1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:31,996 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:31,996 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,997 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:31,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:31,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:32,000 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:32,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024191995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024191995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024191995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:32,004 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:32,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:32,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:32,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:32,004 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:32,006 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,006 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,006 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:32,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,008 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:32,010 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:32,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:32,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:32,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:32,011 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:32,011 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:32,111 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:32,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:32,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:32,114 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:32,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:32,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:32,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:32,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,117 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:32,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:32,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,118 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:32,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:32,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:32,119 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,119 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:32,119 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:32,119 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,121 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,121 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59514edd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:32,123 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:32,123 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,124 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:32,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:32,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:32,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:32,126 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:32,128 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:32,128 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:32,128 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:32,128 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:32,128 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:32,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,131 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:32,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,132 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,132 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,136 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:32,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:32,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:32,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:32,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:32,136 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:32,136 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:32,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:32,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:32,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024192194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024192196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024192203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024192203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024192204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,236 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:32,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:32,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:32,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:32,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:32,244 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:32,244 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:32,244 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:32,244 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:32,244 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,246 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:32,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:32,247 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,247 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:32,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:32,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:32,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:32,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:32,264 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,264 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:32,264 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:32,264 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:32,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:32,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,266 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:32,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f30078[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:32,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:32,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:32,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:32,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:32,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:32,277 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:32,284 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:32,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:32,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:32,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:32,285 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:32,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:32,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:32,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:32,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,289 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,292 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:32,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:32,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:32,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:32,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:32,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:32,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:32,293 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:32,293 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:32,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:32,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:32,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/297746e8ae354be183b8105706ddf551 2023-04-20 20:55:32,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/766bfe21275f43f4b4cc0ed8c7b2df65 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/766bfe21275f43f4b4cc0ed8c7b2df65 2023-04-20 20:55:32,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/766bfe21275f43f4b4cc0ed8c7b2df65, entries=250, sequenceid=550, filesize=39.2 K 2023-04-20 20:55:32,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/be789d1bea9846449c6eb0faf8ae85db as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/be789d1bea9846449c6eb0faf8ae85db 2023-04-20 20:55:32,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/be789d1bea9846449c6eb0faf8ae85db, entries=150, sequenceid=550, filesize=11.9 K 2023-04-20 20:55:32,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/297746e8ae354be183b8105706ddf551 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/297746e8ae354be183b8105706ddf551 2023-04-20 20:55:32,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/297746e8ae354be183b8105706ddf551, entries=150, sequenceid=550, filesize=11.9 K 2023-04-20 20:55:32,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~198.89 KB/203664, currentSize=107.34 KB/109920 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 515ms, sequenceid=550, compaction requested=true 2023-04-20 20:55:32,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:32,356 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:32,356 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:32,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:32,357 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:32,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:32,359 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:32,360 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:32,360 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,360 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0405c64a28d740f6bfe593698628672b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2f6912366c8e472f9e4cc51595bb4125, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/be789d1bea9846449c6eb0faf8ae85db] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=33.7 K 2023-04-20 20:55:32,360 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85618 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:32,360 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:32,360 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,360 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/622acff34b984d3a92a17a93a89ac0d1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/862e2f606fb049ea9d6803ae567af72b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/766bfe21275f43f4b4cc0ed8c7b2df65] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=83.6 K 2023-04-20 20:55:32,360 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0405c64a28d740f6bfe593698628672b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682024129705 2023-04-20 20:55:32,361 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2f6912366c8e472f9e4cc51595bb4125, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682024130004 2023-04-20 20:55:32,361 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 622acff34b984d3a92a17a93a89ac0d1, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682024129705 2023-04-20 20:55:32,362 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting be789d1bea9846449c6eb0faf8ae85db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1682024131826 2023-04-20 20:55:32,362 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 862e2f606fb049ea9d6803ae567af72b, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682024130004 2023-04-20 20:55:32,365 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 766bfe21275f43f4b4cc0ed8c7b2df65, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1682024131824 2023-04-20 20:55:32,373 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#339 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:32,386 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#338 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:32,393 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:32,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:32,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:32,396 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/025c149e6d3845bb8f156d6c982d4221 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/025c149e6d3845bb8f156d6c982d4221 2023-04-20 20:55:32,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:32,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:32,397 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:32,397 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:32,397 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:32,397 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:32,398 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,398 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,399 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:32,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:32,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:32,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:32,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:32,402 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,402 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:32,402 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 20:55:32,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:32,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:32,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:32,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:32,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:32,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:32,403 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 025c149e6d3845bb8f156d6c982d4221(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:32,403 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:32,403 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=13, startTime=1682024132356; duration=0sec 2023-04-20 20:55:32,403 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:32,403 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:32,405 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:32,405 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:32,406 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:32,406 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/5c67c05ef3444465b2b9d835b6a16218, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/113cdf5d3a014f36afa9a7b4905fd796, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/297746e8ae354be183b8105706ddf551] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=33.7 K 2023-04-20 20:55:32,406 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5c67c05ef3444465b2b9d835b6a16218, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682024129705 2023-04-20 20:55:32,406 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 113cdf5d3a014f36afa9a7b4905fd796, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682024130004 2023-04-20 20:55:32,407 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 297746e8ae354be183b8105706ddf551, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1682024131826 2023-04-20 20:55:32,411 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#341 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:32,422 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304208500b0ea26d44848ac894d6e50ebc94a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208500b0ea26d44848ac894d6e50ebc94a 2023-04-20 20:55:32,427 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/1e975e73c1ad443db303f03148902ab5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1e975e73c1ad443db303f03148902ab5 2023-04-20 20:55:32,431 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 1e975e73c1ad443db303f03148902ab5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:32,431 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:32,431 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=13, startTime=1682024132357; duration=0sec 2023-04-20 20:55:32,431 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:32,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:32,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024192517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024192518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024192523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024192526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024192526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024192627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024192627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024192633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024192634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024192635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,834 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=569, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/398aff1c288e4311b851be1f23772508 2023-04-20 20:55:32,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024192832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024192833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024192838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024192839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:32,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:32,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44542 deadline: 1682024192840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024193139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024193141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024193142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024193143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024193151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,209 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/5379e2213f18481b85d3f692af5ec1d2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5379e2213f18481b85d3f692af5ec1d2 2023-04-20 20:55:33,213 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 5379e2213f18481b85d3f692af5ec1d2(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:33,213 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:33,213 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=13, startTime=1682024132356; duration=0sec 2023-04-20 20:55:33,213 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:33,243 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/fec386242957411b8502ee5f9526ac5d 2023-04-20 20:55:33,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44568 deadline: 1682024193645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024193647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024193648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024193648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,654 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e6ab74c3f6274b6aa386c1408ec44e06 2023-04-20 20:55:33,658 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/398aff1c288e4311b851be1f23772508 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/398aff1c288e4311b851be1f23772508 2023-04-20 20:55:33,665 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/398aff1c288e4311b851be1f23772508, entries=150, sequenceid=569, filesize=25.4 K 2023-04-20 20:55:33,666 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/fec386242957411b8502ee5f9526ac5d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/fec386242957411b8502ee5f9526ac5d 2023-04-20 20:55:33,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:33,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024193662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,673 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/fec386242957411b8502ee5f9526ac5d, entries=150, sequenceid=569, filesize=11.9 K 2023-04-20 20:55:33,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e6ab74c3f6274b6aa386c1408ec44e06 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e6ab74c3f6274b6aa386c1408ec44e06 2023-04-20 20:55:33,686 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e6ab74c3f6274b6aa386c1408ec44e06, entries=150, sequenceid=569, filesize=11.9 K 2023-04-20 20:55:33,687 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1285ms, sequenceid=569, compaction requested=false 2023-04-20 20:55:33,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:33,687 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:33,692 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:33,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:33,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b389690[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:33,695 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:33,695 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:33,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:33,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:33,697 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,697 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:33,702 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:33,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:33,706 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:33,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:33,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:33,706 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:33,706 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:33,707 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,708 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:33,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:33,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:33,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:33,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:33,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:33,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:33,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:33,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:33,714 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:33,714 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:33,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:33,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:33,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:33,814 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:33,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:33,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:33,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:33,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:33,821 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:33,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:33,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:33,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:33,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:33,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:33,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:33,825 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,825 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:33,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:33,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:33,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:33,827 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:33,833 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:33,833 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:33,848 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:33,848 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:33,848 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 20:55:33,848 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:33,848 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:33,848 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:33,849 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:33,849 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:33,849 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:33,882 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420cf5049562fe44df18c5c32b9d440684a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420cf5049562fe44df18c5c32b9d440684a 2023-04-20 20:55:33,899 WARN [DataStreamer for file /user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/905053e5ab9947839249a33945f6b985] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-20 20:55:33,900 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=586, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/905053e5ab9947839249a33945f6b985 2023-04-20 20:55:33,915 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/dd64747d220649f6bacedd2911b08f69 2023-04-20 20:55:33,939 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/c6acb3df72374e81a828391d20cec8fe 2023-04-20 20:55:33,944 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/905053e5ab9947839249a33945f6b985 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/905053e5ab9947839249a33945f6b985 2023-04-20 20:55:33,951 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/905053e5ab9947839249a33945f6b985, entries=150, sequenceid=586, filesize=25.4 K 2023-04-20 20:55:33,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/dd64747d220649f6bacedd2911b08f69 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/dd64747d220649f6bacedd2911b08f69 2023-04-20 20:55:33,957 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/dd64747d220649f6bacedd2911b08f69, entries=150, sequenceid=586, filesize=11.9 K 2023-04-20 20:55:33,957 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/c6acb3df72374e81a828391d20cec8fe as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/c6acb3df72374e81a828391d20cec8fe 2023-04-20 20:55:33,961 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/c6acb3df72374e81a828391d20cec8fe, entries=150, sequenceid=586, filesize=11.9 K 2023-04-20 20:55:33,961 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 113ms, sequenceid=586, compaction requested=true 2023-04-20 20:55:33,962 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:33,962 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,964 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:33,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:33,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d20682[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:33,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:33,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:33,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:33,969 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:33,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:33,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:33,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:33,972 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:33,972 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:33,972 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:33,972 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:33,972 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:33,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:33,976 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:33,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:33,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:33,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:33,981 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:33,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:33,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:33,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:33,982 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:33,982 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,082 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:34,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:34,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:34,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:34,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:34,086 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:34,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:34,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:34,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,088 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,088 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,088 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,088 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:34,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:34,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:34,090 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,090 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:34,090 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:34,090 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:34,092 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:34,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33a69be3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:34,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:34,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,096 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:34,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:34,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:34,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:34,101 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:34,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:34,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:34,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:34,101 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:34,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,107 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:34,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,107 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:34,107 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,207 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:34,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:34,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:34,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:34,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:34,211 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:34,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:34,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:34,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,214 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,214 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,214 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:34,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:34,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:34,215 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:34,216 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,216 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:34,216 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:34,216 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:34,217 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:34,218 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,218 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:34,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3fbd2f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:34,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:34,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,221 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:34,221 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,221 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:34,222 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:34,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:34,225 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:34,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:34,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:34,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:34,225 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:34,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:34,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:34,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,238 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:34,238 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,338 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:34,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:34,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:34,342 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:34,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,342 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:34,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:34,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,346 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,347 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:34,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:34,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:34,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:34,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:34,351 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:34,353 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,354 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,354 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:34,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@355c3cee[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:34,356 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:34,356 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:34,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:34,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,359 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:34,362 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:34,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:34,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:34,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:34,363 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:34,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,365 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,369 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:34,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,370 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,370 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:34,370 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:34,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,470 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:34,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:34,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:34,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:34,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:34,474 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:34,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:34,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:34,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:34,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:34,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:34,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:34,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:34,480 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:34,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cacad1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:34,484 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:34,484 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,486 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:34,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:34,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:34,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:34,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:34,491 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:34,491 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:34,491 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:34,491 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:34,491 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:34,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,495 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:34,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:34,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,499 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,498 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:34,499 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,499 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,599 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:34,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:34,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:34,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:34,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:34,602 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:34,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:34,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:34,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,606 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,607 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:34,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:34,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:34,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:34,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:34,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:34,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:34,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:34,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:34,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:34,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:34,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:34,612 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42343808[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:34,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:34,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,616 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:34,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:34,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:34,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:34,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:34,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:34,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:34,622 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:34,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:34,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:34,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:34,622 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:34,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,624 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,629 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:34,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:34,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,630 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,630 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:34,630 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:34,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:55:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:34,730 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:34,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:34,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:34,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:34,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:34,736 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:34,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:34,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:34,737 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042022017daf7fbd441195ecb12d587c0fe4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042022017daf7fbd441195ecb12d587c0fe4 2023-04-20 20:55:34,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,738 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:34,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:34,767 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:34,767 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:34,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,769 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=602, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/9518b65d187744e380990ea5906db3c7 2023-04-20 20:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c533a4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:34,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:34,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,773 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:34,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:34,773 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:34,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:34,775 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,775 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,776 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,776 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:34,779 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:34,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:34,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:34,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:34,779 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:34,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024194778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024134735, max timestamp=1682024134781Num uniques 859; 2023-04-20 20:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024134735, max timestamp=1682024134781Num uniques 859; 2023-04-20 20:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024134735, max timestamp=1682024134781Num uniques 859; 2023-04-20 20:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,783 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,785 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:34,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:34,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,789 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:34,789 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024194804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024194809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024194810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024194816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024194883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,889 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:34,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:34,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:34,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:34,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:34,892 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:34,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,893 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:34,893 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:34,893 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,893 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,895 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:34,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:34,896 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,897 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:34,897 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:34,897 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:34,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:34,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:34,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:34,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:34,898 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,898 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31aad8be[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:34,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:34,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:34,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:34,901 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:34,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:34,903 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,903 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:34,905 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:34,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:34,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:34,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:34,905 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:34,907 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,907 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,907 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:34,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:34,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:34,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:34,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:34,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:34,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:34,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:34,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:34,912 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:34,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:34,912 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:34,912 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:34,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024194917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024194920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024194921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:34,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:34,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024194921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,012 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:35,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:35,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:35,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:35,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:35,019 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:35,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:35,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:35,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:35,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,021 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:35,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:35,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:35,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:35,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:35,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:35,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:35,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:35,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:35,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:35,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,022 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:35,023 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:35,023 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,023 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:35,023 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:35,023 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,023 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:35,023 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:35,025 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:35,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:35,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:35,027 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:35,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22250018[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:35,027 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:35,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,028 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:35,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:35,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:35,030 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:35,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:35,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:35,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:35,032 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:35,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:35,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:35,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:35,032 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:35,034 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,034 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:35,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,035 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,036 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:35,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:35,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:35,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:35,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:35,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:35,046 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:35,046 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:35,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:35,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:35,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024195086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024195127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024195127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024195128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024195128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,146 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:35,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:35,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:35,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:35,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:35,149 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:35,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:35,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:35,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:35,150 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,150 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,155 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,155 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:35,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:35,155 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,155 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:35,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:35,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:35,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:35,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:35,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:35,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:35,156 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,156 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:35,157 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:35,157 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,157 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:35,157 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:35,157 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,157 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:35,157 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:35,158 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:35,159 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,159 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:35,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:35,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41b2882c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:35,161 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:35,161 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,162 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:35,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:35,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:35,162 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,162 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:35,163 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,163 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:35,163 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:35,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:35,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:35,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:35,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:35,165 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:35,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:35,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:35,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:35,165 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:35,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:35,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:35,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:35,168 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,169 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:35,171 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:35,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:35,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:35,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:35,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:35,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:35,172 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:35,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:35,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:35,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,172 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:35,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/684e759f4bd946898253017dc0ae5078 2023-04-20 20:55:35,196 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e5d5df441a9a4ce385c12844ed88b580 2023-04-20 20:55:35,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/9518b65d187744e380990ea5906db3c7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9518b65d187744e380990ea5906db3c7 2023-04-20 20:55:35,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9518b65d187744e380990ea5906db3c7, entries=200, sequenceid=602, filesize=32.3 K 2023-04-20 20:55:35,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/684e759f4bd946898253017dc0ae5078 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/684e759f4bd946898253017dc0ae5078 2023-04-20 20:55:35,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/684e759f4bd946898253017dc0ae5078, entries=150, sequenceid=602, filesize=11.9 K 2023-04-20 20:55:35,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e5d5df441a9a4ce385c12844ed88b580 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e5d5df441a9a4ce385c12844ed88b580 2023-04-20 20:55:35,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e5d5df441a9a4ce385c12844ed88b580, entries=150, sequenceid=602, filesize=11.9 K 2023-04-20 20:55:35,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 486ms, sequenceid=602, compaction requested=true 2023-04-20 20:55:35,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:35,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:35,211 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:35,211 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:35,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:35,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:35,213 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:35,213 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:35,213 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,213 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/025c149e6d3845bb8f156d6c982d4221, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/fec386242957411b8502ee5f9526ac5d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/dd64747d220649f6bacedd2911b08f69, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/684e759f4bd946898253017dc0ae5078] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=48.1 K 2023-04-20 20:55:35,213 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111770 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:35,213 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:35,213 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,213 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5379e2213f18481b85d3f692af5ec1d2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/398aff1c288e4311b851be1f23772508, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/905053e5ab9947839249a33945f6b985, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9518b65d187744e380990ea5906db3c7] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=109.2 K 2023-04-20 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 025c149e6d3845bb8f156d6c982d4221, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1682024131826 2023-04-20 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5379e2213f18481b85d3f692af5ec1d2, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1682024131826 2023-04-20 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting fec386242957411b8502ee5f9526ac5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1682024131874 2023-04-20 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 398aff1c288e4311b851be1f23772508, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1682024131874 2023-04-20 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting dd64747d220649f6bacedd2911b08f69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682024132516 2023-04-20 20:55:35,215 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 905053e5ab9947839249a33945f6b985, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682024132516 2023-04-20 20:55:35,215 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 684e759f4bd946898253017dc0ae5078, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682024134702 2023-04-20 20:55:35,215 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9518b65d187744e380990ea5906db3c7, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682024134702 2023-04-20 20:55:35,228 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#350 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:35,239 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#351 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:35,272 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:35,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:35,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:35,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:35,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:35,278 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:35,278 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:35,278 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:35,278 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:35,279 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,279 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,281 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:35,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:35,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:35,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:35,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:35,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:35,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:35,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:35,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:35,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:35,284 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-20 20:55:35,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:35,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:35,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:35,285 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:35,285 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:35,285 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:35,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:35,403 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:35,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024195438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024195438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024195440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024195442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024195443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024195544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024195554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024195555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,645 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/3fd5060d50894bf48c35fd3033e1b4d5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3fd5060d50894bf48c35fd3033e1b4d5 2023-04-20 20:55:35,651 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 3fd5060d50894bf48c35fd3033e1b4d5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:35,651 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:35,651 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=12, startTime=1682024135211; duration=0sec 2023-04-20 20:55:35,651 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:35,651 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:35,652 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:35,652 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:35,652 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:35,653 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1e975e73c1ad443db303f03148902ab5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e6ab74c3f6274b6aa386c1408ec44e06, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/c6acb3df72374e81a828391d20cec8fe, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e5d5df441a9a4ce385c12844ed88b580] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=48.1 K 2023-04-20 20:55:35,653 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1e975e73c1ad443db303f03148902ab5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1682024131826 2023-04-20 20:55:35,654 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e6ab74c3f6274b6aa386c1408ec44e06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1682024131874 2023-04-20 20:55:35,654 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c6acb3df72374e81a828391d20cec8fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682024132516 2023-04-20 20:55:35,655 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e5d5df441a9a4ce385c12844ed88b580, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682024134702 2023-04-20 20:55:35,665 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:35,691 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/a5e8236dec584c779316346381708de6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a5e8236dec584c779316346381708de6 2023-04-20 20:55:35,704 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into a5e8236dec584c779316346381708de6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:35,704 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:35,704 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=12, startTime=1682024135211; duration=0sec 2023-04-20 20:55:35,704 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:35,706 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/6707072e9f2c4835b0c14f494ec060b3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6707072e9f2c4835b0c14f494ec060b3 2023-04-20 20:55:35,712 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 6707072e9f2c4835b0c14f494ec060b3(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:35,712 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:35,712 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=12, startTime=1682024135211; duration=0sec 2023-04-20 20:55:35,712 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:35,718 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420b261d98ee1d84ce5bb75fa6d5b0d59e1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b261d98ee1d84ce5bb75fa6d5b0d59e1 2023-04-20 20:55:35,725 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=636, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/9f91c35b5ac94621b576402055f2a44f 2023-04-20 20:55:35,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024195751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024195760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024195762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024195947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:35,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:35,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024195948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024196059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024196069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024196070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,134 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/b5d15e4103514444b64bb745d7e8fb4d 2023-04-20 20:55:36,543 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/0bcb32adefd248b0bc5ab9c0a7de4ef7 2023-04-20 20:55:36,548 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/9f91c35b5ac94621b576402055f2a44f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9f91c35b5ac94621b576402055f2a44f 2023-04-20 20:55:36,552 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9f91c35b5ac94621b576402055f2a44f, entries=150, sequenceid=636, filesize=25.4 K 2023-04-20 20:55:36,552 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/b5d15e4103514444b64bb745d7e8fb4d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/b5d15e4103514444b64bb745d7e8fb4d 2023-04-20 20:55:36,556 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/b5d15e4103514444b64bb745d7e8fb4d, entries=150, sequenceid=636, filesize=11.9 K 2023-04-20 20:55:36,557 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/0bcb32adefd248b0bc5ab9c0a7de4ef7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0bcb32adefd248b0bc5ab9c0a7de4ef7 2023-04-20 20:55:36,560 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0bcb32adefd248b0bc5ab9c0a7de4ef7, entries=150, sequenceid=636, filesize=11.9 K 2023-04-20 20:55:36,561 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1277ms, sequenceid=636, compaction requested=false 2023-04-20 20:55:36,561 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:36,561 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:36,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:36,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:36,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:36,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:36,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:36,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d26cbad[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:36,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:36,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,569 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:36,569 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:36,570 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:36,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:36,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,571 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:36,571 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,572 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:36,573 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:55:36,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:36,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:36,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:36,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:36,574 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:36,574 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:36,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:36,576 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:36,576 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:36,577 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:36,577 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:36,577 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:36,579 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,579 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,579 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,584 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:36,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:36,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:36,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:36,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:36,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:36,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,589 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:36,589 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,589 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:36,589 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,589 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,589 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,589 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,597 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042054690f29d76f493f9dae187cae4bac73 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042054690f29d76f493f9dae187cae4bac73 2023-04-20 20:55:36,608 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=651, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/dd5726e63c0f4b5fb0fef382523b51b9 2023-04-20 20:55:36,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024196621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024196622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024196622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,689 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:36,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:36,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:36,693 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:36,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:36,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:36,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:36,694 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,694 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,696 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:36,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:36,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:36,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:36,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:36,699 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:36,699 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:36,699 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:36,699 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:36,699 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:36,699 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:36,699 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:36,699 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:36,699 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,699 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:36,699 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:36,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,701 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,701 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,702 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:36,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a2e9d9f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:36,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:36,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,704 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:36,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:36,704 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:36,704 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,704 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:36,706 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:36,706 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,706 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,706 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:36,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:36,710 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:36,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:36,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:36,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:36,710 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:36,712 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,712 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,712 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,716 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:36,716 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:36,716 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:36,716 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:36,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:36,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,717 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,717 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,717 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,717 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,717 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,717 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,718 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:36,718 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:36,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:36,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024196733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024196734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024196734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,818 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:36,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:36,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:36,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:36,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:36,822 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:36,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:36,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:36,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:36,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:36,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:36,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:36,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:36,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:36,826 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:36,826 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:36,826 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:36,826 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:36,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:36,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60016502[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:36,835 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:36,835 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,836 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:36,836 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,836 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:36,838 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:36,838 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,838 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,838 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:36,840 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:36,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:36,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:36,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:36,840 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:36,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,842 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,842 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,844 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:36,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:36,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:36,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:36,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:36,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,847 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:36,848 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:36,847 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,847 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,848 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,848 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:36,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024196937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024196940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024196940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,948 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:36,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:36,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:36,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:36,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:36,953 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:36,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:36,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:36,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:36,954 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,954 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,956 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:36,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:36,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:36,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:36,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:36,958 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:36,958 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:36,958 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:36,958 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:36,959 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:36,959 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,959 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:36,959 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:36,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024196956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:36,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352762c6[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:36,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:36,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:36,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024196957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,963 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:36,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:36,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:36,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:36,972 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:36,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:36,975 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:36,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:36,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:36,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:36,975 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:36,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:36,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:36,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:36,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:36,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:36,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:36,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,981 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:36,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:36,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:36,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:36,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:36,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:36,982 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:36,982 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:36,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:37,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/e3d7b0d64be142d4ba1540700c212328 2023-04-20 20:55:37,083 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:37,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:37,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:37,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:37,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:37,086 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:37,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:37,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:37,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,088 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:37,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:37,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,089 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:37,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:37,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:37,092 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,092 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:37,092 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:37,092 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:37,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee7bc6c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:37,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:37,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:37,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:37,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:37,098 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,098 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:37,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:37,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:37,104 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:37,106 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:37,110 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:37,110 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:37,110 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:37,110 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:37,110 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:37,114 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,114 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:37,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,114 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:37,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,116 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:37,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:37,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:37,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:37,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:37,119 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:37,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,119 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:37,120 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:37,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:37,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:37,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:37,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:37,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,120 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,220 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:37,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:37,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:37,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:37,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:37,224 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:37,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:37,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:37,228 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,228 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:37,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:37,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:37,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:37,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:37,232 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:37,233 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,233 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:37,233 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:37,233 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:37,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,235 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,235 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,235 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10bb0a4a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:37,236 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:37,236 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,238 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:37,238 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:37,238 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:37,238 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,238 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:37,242 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:37,242 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:37,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:37,245 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:37,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:37,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:37,245 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:37,245 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:37,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,248 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:37,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,249 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,250 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024197242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024197246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:37,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024197247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:37,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:37,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:37,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:37,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:37,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:37,254 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:37,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,255 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:37,255 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:37,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,355 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:37,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:37,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:37,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:37,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:37,358 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:37,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:37,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:37,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,360 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:37,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:37,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:37,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:37,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:37,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:37,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:37,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:37,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:37,362 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,362 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:37,362 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:37,362 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:37,365 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,365 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,365 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,365 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b79542a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:37,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:37,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,368 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,368 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:37,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,369 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:37,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:37,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:37,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:37,372 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:37,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:37,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:37,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:37,372 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:37,374 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,374 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,375 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:37,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:37,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:37,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:37,375 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:37,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:37,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,380 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,380 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:37,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:37,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:37,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:37,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:37,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:37,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:37,382 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:37,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:37,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:37,383 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:37,383 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:37,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e79686ffcdc04a0eb999772553557ed1 2023-04-20 20:55:37,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/dd5726e63c0f4b5fb0fef382523b51b9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dd5726e63c0f4b5fb0fef382523b51b9 2023-04-20 20:55:37,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dd5726e63c0f4b5fb0fef382523b51b9, entries=200, sequenceid=651, filesize=32.3 K 2023-04-20 20:55:37,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/e3d7b0d64be142d4ba1540700c212328 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/e3d7b0d64be142d4ba1540700c212328 2023-04-20 20:55:37,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/e3d7b0d64be142d4ba1540700c212328, entries=150, sequenceid=651, filesize=11.9 K 2023-04-20 20:55:37,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/e79686ffcdc04a0eb999772553557ed1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e79686ffcdc04a0eb999772553557ed1 2023-04-20 20:55:37,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e79686ffcdc04a0eb999772553557ed1, entries=150, sequenceid=651, filesize=11.9 K 2023-04-20 20:55:37,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 878ms, sequenceid=651, compaction requested=true 2023-04-20 20:55:37,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:37,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:37,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:37,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 20:55:37,452 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:37,452 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:37,454 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:37,454 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:37,454 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,454 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6707072e9f2c4835b0c14f494ec060b3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9f91c35b5ac94621b576402055f2a44f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dd5726e63c0f4b5fb0fef382523b51b9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=83.8 K 2023-04-20 20:55:37,454 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:37,454 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:37,454 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,454 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3fd5060d50894bf48c35fd3033e1b4d5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/b5d15e4103514444b64bb745d7e8fb4d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/e3d7b0d64be142d4ba1540700c212328] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=36.4 K 2023-04-20 20:55:37,454 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6707072e9f2c4835b0c14f494ec060b3, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682024134702 2023-04-20 20:55:37,455 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3fd5060d50894bf48c35fd3033e1b4d5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682024134702 2023-04-20 20:55:37,455 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9f91c35b5ac94621b576402055f2a44f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1682024134798 2023-04-20 20:55:37,455 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b5d15e4103514444b64bb745d7e8fb4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1682024134798 2023-04-20 20:55:37,456 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting dd5726e63c0f4b5fb0fef382523b51b9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1682024135408 2023-04-20 20:55:37,456 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e3d7b0d64be142d4ba1540700c212328, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1682024135408 2023-04-20 20:55:37,466 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:37,470 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#359 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:37,483 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:37,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:37,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:37,491 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/4adb9ea075914323872eaaee4e09c724 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/4adb9ea075914323872eaaee4e09c724 2023-04-20 20:55:37,495 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 4adb9ea075914323872eaaee4e09c724(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:37,496 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:37,496 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=13, startTime=1682024137451; duration=0sec 2023-04-20 20:55:37,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:37,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:37,496 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:37,496 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:37,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:37,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:37,496 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:37,497 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,497 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,497 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:37,498 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:37,498 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:37,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:37,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:37,498 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a5e8236dec584c779316346381708de6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0bcb32adefd248b0bc5ab9c0a7de4ef7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e79686ffcdc04a0eb999772553557ed1] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=36.4 K 2023-04-20 20:55:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,499 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a5e8236dec584c779316346381708de6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682024134702 2023-04-20 20:55:37,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:37,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:37,500 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0bcb32adefd248b0bc5ab9c0a7de4ef7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1682024134798 2023-04-20 20:55:37,501 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e79686ffcdc04a0eb999772553557ed1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1682024135408 2023-04-20 20:55:37,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:37,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:37,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:37,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:37,504 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:37,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:37,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:37,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:37,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:37,512 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 20:55:37,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:37,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:37,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:37,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:37,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:37,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:37,526 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#361 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:37,532 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/b382eb49c56746889a6a7d5128eb6646 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/b382eb49c56746889a6a7d5128eb6646 2023-04-20 20:55:37,539 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into b382eb49c56746889a6a7d5128eb6646(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:37,539 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:37,539 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=13, startTime=1682024137451; duration=0sec 2023-04-20 20:55:37,539 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:37,584 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420e73ea9943e1443efbd7fda9a9a0b25d5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e73ea9943e1443efbd7fda9a9a0b25d5 2023-04-20 20:55:37,607 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=672, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d 2023-04-20 20:55:37,626 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/3736ee2195794f038581c953252c19c5 2023-04-20 20:55:37,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:37,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:37,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024197778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024197780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024197788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024197889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024197890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:37,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024197901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:37,958 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/7b30b2a360ab4c4984c66cb90e99a097 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/7b30b2a360ab4c4984c66cb90e99a097 2023-04-20 20:55:37,963 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 7b30b2a360ab4c4984c66cb90e99a097(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:37,964 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:37,964 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=13, startTime=1682024137452; duration=0sec 2023-04-20 20:55:37,964 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:38,042 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/0b233e19ad404b47a372a7a9b5c005bf 2023-04-20 20:55:38,047 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d 2023-04-20 20:55:38,051 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d, entries=150, sequenceid=672, filesize=25.4 K 2023-04-20 20:55:38,052 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/3736ee2195794f038581c953252c19c5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3736ee2195794f038581c953252c19c5 2023-04-20 20:55:38,055 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3736ee2195794f038581c953252c19c5, entries=150, sequenceid=672, filesize=11.9 K 2023-04-20 20:55:38,056 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/0b233e19ad404b47a372a7a9b5c005bf as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0b233e19ad404b47a372a7a9b5c005bf 2023-04-20 20:55:38,059 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0b233e19ad404b47a372a7a9b5c005bf, entries=150, sequenceid=672, filesize=11.9 K 2023-04-20 20:55:38,060 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 548ms, sequenceid=672, compaction requested=false 2023-04-20 20:55:38,060 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:38,060 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:38,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:38,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:38,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:38,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,062 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@754628d1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:38,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:38,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:38,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:38,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:38,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:38,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:38,069 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:38,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:38,071 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:38,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:38,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:38,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:38,071 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:38,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,075 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,077 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,079 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:38,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:38,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,079 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:38,080 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:38,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:38,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 20:55:38,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:38,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:38,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:38,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:38,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:38,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:38,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024198148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024198152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024198153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,180 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:38,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:38,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:38,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:38,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:38,183 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:38,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:38,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:38,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,184 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,184 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:38,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:38,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:38,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:38,188 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,188 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:38,188 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:38,188 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:38,190 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,191 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:38,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:38,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46f1eec9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:38,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,193 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:38,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:38,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:38,193 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,193 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:38,194 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:38,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:38,198 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:38,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:38,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:38,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:38,198 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:38,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,210 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:38,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,210 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:38,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,210 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:38,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:38,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024198257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024198264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024198264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,311 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:38,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:38,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:38,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:38,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:38,316 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:38,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:38,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:38,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,318 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:38,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:38,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:38,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:38,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:38,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:38,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:38,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:38,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:38,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:38,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:38,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:38,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,323 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:38,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed05507[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:38,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:38,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:38,329 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,329 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:38,330 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,330 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:38,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:38,335 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:38,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:38,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:38,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:38,335 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:38,337 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,338 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,338 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,338 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,340 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:38,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:38,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,343 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:38,343 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:38,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,344 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,344 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,344 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:38,443 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:38,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:38,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:38,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:38,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:38,447 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:38,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:38,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:38,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,450 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,450 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,450 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:38,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:38,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:38,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:38,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:38,453 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,454 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,454 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:38,455 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f3998f3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:38,455 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,457 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:38,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:38,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:38,461 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:38,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:38,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:38,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:38,461 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:38,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,464 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,466 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,468 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,468 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,468 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,468 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:38,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:38,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,468 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:38,468 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,468 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,469 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:38,469 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:38,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024198464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024198471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024198471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,527 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304205b9db0f63df74ea8a41038a6c1f90f5f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205b9db0f63df74ea8a41038a6c1f90f5f 2023-04-20 20:55:38,532 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=688, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/822287b7620646a69decbe98128f619f 2023-04-20 20:55:38,569 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:38,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:38,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:38,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:38,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:38,579 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:38,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:38,579 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:38,580 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,580 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,581 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:38,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:38,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:38,584 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,584 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:38,584 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:38,584 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,584 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:38,586 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:38,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bfd2f65[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:38,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:38,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,590 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:38,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:38,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:38,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,591 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:38,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:38,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:38,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:38,592 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:38,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:38,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:38,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:38,592 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:38,594 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,594 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,595 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:38,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:38,604 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:38,604 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:38,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,704 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:38,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:38,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:38,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:38,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:38,708 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:38,708 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,708 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:38,708 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:38,709 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,709 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,710 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:38,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:38,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:38,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:38,712 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,712 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:38,712 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:38,712 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:38,713 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:38,714 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,714 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,714 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:38,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b30175[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:38,716 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:38,716 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,719 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,719 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:38,719 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:38,719 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:38,720 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,720 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:38,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:38,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:38,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:38,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:38,737 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:38,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:38,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:38,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:38,737 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:38,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,750 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:38,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,753 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:38,754 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:38,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:38,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024198773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024198776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024198777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,854 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:38,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:38,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:38,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:38,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:38,861 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:38,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:38,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:38,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,863 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:38,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:38,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:38,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:38,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:38,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:38,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:38,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:38,883 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:38,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec70249[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:38,885 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:38,885 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,888 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:38,888 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:38,888 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:38,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:38,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:38,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,889 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,889 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:38,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:38,892 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:38,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:38,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:38,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:38,892 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:38,894 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,894 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,894 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:38,895 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:38,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:38,914 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:38,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:38,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:38,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:38,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:38,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:38,915 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:38,915 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:38,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:38,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/584759f3fc484a3e864fe819e4d712db 2023-04-20 20:55:38,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44542 deadline: 1682024198979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:38,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44530 deadline: 1682024198983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:38,991 DEBUG [Thread-2177] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4185 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:38,992 DEBUG [Thread-2179] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:55:39,015 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:39,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:39,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:39,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:39,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:39,019 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:39,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:39,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:39,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,019 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,021 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:39,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:39,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:39,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:39,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:39,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:39,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:39,044 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,044 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:39,044 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:39,044 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:39,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:39,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:39,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:39,048 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:39,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d4aeb62[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:39,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:39,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,052 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:39,053 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,053 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:39,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:39,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:39,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:39,057 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:39,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:39,057 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:39,060 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,060 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,060 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,062 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:39,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,071 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:39,071 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:39,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,172 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:39,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:39,176 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:39,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:39,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:39,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,179 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,179 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:39,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:39,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:39,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:39,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:39,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:39,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:39,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:39,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:39,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:39,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5e73a9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:39,189 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:39,189 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:39,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:39,194 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:39,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:39,197 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:39,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:39,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:39,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:39,198 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:39,200 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:39,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:39,207 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:39,207 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:39,206 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,207 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:39,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024199282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:39,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024199283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:39,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024199287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,307 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:39,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:39,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:39,310 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:39,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:39,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:39,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:39,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:39,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:39,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:39,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:39,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:39,315 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,315 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:39,315 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:39,315 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:39,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:39,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:39,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,318 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75cc699e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:39,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:39,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,321 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:39,321 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,321 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:39,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:39,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:39,323 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:39,325 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:39,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:39,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:39,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:39,326 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:39,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,330 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,331 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:39,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:39,333 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:39,333 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:39,333 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:39,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,434 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:39,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:39,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:39,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:39,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:39,437 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:39,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:39,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:39,437 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,438 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:39,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:39,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:39,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:39,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:39,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:39,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:39,452 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:39,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,454 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:39,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739ff824[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:39,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:39,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:39,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:39,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:39,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:39,461 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:39,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:39,463 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:39,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:39,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:39,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:39,463 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:39,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:39,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:39,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,466 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:39,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/975d0a73e4c1435f938d5f7b37d83dd4 2023-04-20 20:55:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:39,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/822287b7620646a69decbe98128f619f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/822287b7620646a69decbe98128f619f 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:39,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:39,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:39,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:39,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,474 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:39,474 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:39,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:39,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/822287b7620646a69decbe98128f619f, entries=250, sequenceid=688, filesize=39.2 K 2023-04-20 20:55:39,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/584759f3fc484a3e864fe819e4d712db as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/584759f3fc484a3e864fe819e4d712db 2023-04-20 20:55:39,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/584759f3fc484a3e864fe819e4d712db, entries=150, sequenceid=688, filesize=11.9 K 2023-04-20 20:55:39,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/975d0a73e4c1435f938d5f7b37d83dd4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/975d0a73e4c1435f938d5f7b37d83dd4 2023-04-20 20:55:39,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/975d0a73e4c1435f938d5f7b37d83dd4, entries=150, sequenceid=688, filesize=11.9 K 2023-04-20 20:55:39,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1387ms, sequenceid=688, compaction requested=true 2023-04-20 20:55:39,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:39,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:39,490 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:39,490 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:39,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:39,491 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:39,492 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93006 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:39,492 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:39,492 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,492 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/b382eb49c56746889a6a7d5128eb6646, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/822287b7620646a69decbe98128f619f] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=90.8 K 2023-04-20 20:55:39,492 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:39,492 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:39,492 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,493 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/4adb9ea075914323872eaaee4e09c724, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3736ee2195794f038581c953252c19c5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/584759f3fc484a3e864fe819e4d712db] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=36.5 K 2023-04-20 20:55:39,492 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b382eb49c56746889a6a7d5128eb6646, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1682024135408 2023-04-20 20:55:39,493 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4adb9ea075914323872eaaee4e09c724, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1682024135408 2023-04-20 20:55:39,493 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5d1bd5c4e96c4ce9b57534bf2d3c1d4d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682024136603 2023-04-20 20:55:39,493 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3736ee2195794f038581c953252c19c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682024136603 2023-04-20 20:55:39,493 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 822287b7620646a69decbe98128f619f, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682024137776 2023-04-20 20:55:39,494 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 584759f3fc484a3e864fe819e4d712db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682024137786 2023-04-20 20:55:39,500 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:39,503 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#368 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:39,574 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:39,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:39,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:39,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:39,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:39,582 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:39,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:39,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:39,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,584 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:39,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:39,584 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:39,584 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:39,585 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:39,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:39,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:39,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:39,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:39,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:39,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:39,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:39,586 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:39,586 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:39,586 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:39,920 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/f727de012faa42128a308bc7c2f2116c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f727de012faa42128a308bc7c2f2116c 2023-04-20 20:55:39,924 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into f727de012faa42128a308bc7c2f2116c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:39,924 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:39,924 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=13, startTime=1682024139490; duration=0sec 2023-04-20 20:55:39,924 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:39,924 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:39,925 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:39,925 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:39,925 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:39,925 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/7b30b2a360ab4c4984c66cb90e99a097, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0b233e19ad404b47a372a7a9b5c005bf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/975d0a73e4c1435f938d5f7b37d83dd4] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=36.5 K 2023-04-20 20:55:39,926 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7b30b2a360ab4c4984c66cb90e99a097, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1682024135408 2023-04-20 20:55:39,926 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0b233e19ad404b47a372a7a9b5c005bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682024136603 2023-04-20 20:55:39,926 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 975d0a73e4c1435f938d5f7b37d83dd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682024137786 2023-04-20 20:55:39,938 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:39,952 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/cac6516794eb4e8490543c70e446ea9d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/cac6516794eb4e8490543c70e446ea9d 2023-04-20 20:55:39,956 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into cac6516794eb4e8490543c70e446ea9d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:39,956 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:39,957 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=13, startTime=1682024139491; duration=0sec 2023-04-20 20:55:39,957 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:40,026 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304209a5cb414a7c44c22927b67669a82be70 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209a5cb414a7c44c22927b67669a82be70 2023-04-20 20:55:40,057 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=706, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/9ad4b4fe7fb346e7a84606de7d68c21d 2023-04-20 20:55:40,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:40,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:40,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024200314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024200314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024200319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,339 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/640498cabadc40848ec7883a2787ff76 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/640498cabadc40848ec7883a2787ff76 2023-04-20 20:55:40,347 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 640498cabadc40848ec7883a2787ff76(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:40,347 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:40,347 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=13, startTime=1682024139490; duration=0sec 2023-04-20 20:55:40,347 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:40,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024200420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024200420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024200421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,478 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/f3573ae1cd4e4376904e39f3a9fb2437 2023-04-20 20:55:40,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024200626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024200627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024200627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,891 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/82fd61c99763425998693540bf1d7121 2023-04-20 20:55:40,894 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/9ad4b4fe7fb346e7a84606de7d68c21d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9ad4b4fe7fb346e7a84606de7d68c21d 2023-04-20 20:55:40,902 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9ad4b4fe7fb346e7a84606de7d68c21d, entries=150, sequenceid=706, filesize=25.4 K 2023-04-20 20:55:40,902 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/f3573ae1cd4e4376904e39f3a9fb2437 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f3573ae1cd4e4376904e39f3a9fb2437 2023-04-20 20:55:40,906 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f3573ae1cd4e4376904e39f3a9fb2437, entries=150, sequenceid=706, filesize=11.9 K 2023-04-20 20:55:40,907 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/82fd61c99763425998693540bf1d7121 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/82fd61c99763425998693540bf1d7121 2023-04-20 20:55:40,910 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/82fd61c99763425998693540bf1d7121, entries=150, sequenceid=706, filesize=11.9 K 2023-04-20 20:55:40,910 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1324ms, sequenceid=706, compaction requested=false 2023-04-20 20:55:40,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:40,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:40,916 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,916 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:40,917 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,917 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:40,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:40,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69140cce[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:40,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:40,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,920 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,920 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:40,920 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:40,920 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:40,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:40,923 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:40,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:40,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:40,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:40,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:40,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:40,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:40,926 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:40,926 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:40,926 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:40,926 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:40,926 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:40,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:40,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:40,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:40,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:40,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:40,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:40,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:40,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:40,935 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:40,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:40,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:40,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:40,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:40,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,936 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:40,936 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:40,936 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:55:40,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:40,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:40,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:40,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:40,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:40,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:40,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:40,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:40,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:40,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,937 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:40,937 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:40,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:40,957 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420e2a57395aebe44f888385dec69cc3d92 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e2a57395aebe44f888385dec69cc3d92 2023-04-20 20:55:40,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024200971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024200972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:40,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:40,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024200974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,036 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:41,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:41,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:41,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:41,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:41,039 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:41,039 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:41,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:41,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:41,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:41,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:41,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:41,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:41,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:41,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:41,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:41,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:41,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:41,043 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:41,044 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:41,044 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:41,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:41,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:41,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:41,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:41,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:41,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457bf97e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:41,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,049 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:41,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:41,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:41,050 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:41,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:41,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:41,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:41,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:41,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:41,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:41,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:41,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:41,054 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:41,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:41,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:41,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:41,054 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:41,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:41,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:41,058 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:41,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,062 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:41,062 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:41,062 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:41,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:41,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:41,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024201079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024201079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024201082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,162 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:41,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:41,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:41,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:41,166 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:41,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:41,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:41,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:41,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:41,169 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:41,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:41,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:41,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:41,175 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:41,175 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. started... 2023-04-20 20:55:41,175 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. as already flushing 2023-04-20 20:55:41,175 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:41,177 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:41,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:41,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:41,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53edf8d6[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:41,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,181 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,181 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,181 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:41,181 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:41,181 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:41,181 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:41,183 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:41,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:41,185 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:41,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:41,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:41,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:41,185 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:41,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:41,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:41,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:41,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:41,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:41,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:41,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,192 DEBUG [Thread-2187] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:41,192 DEBUG [Thread-2187] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:41,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:41,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:41,224 DEBUG [Thread-2192] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:62055 2023-04-20 20:55:41,225 DEBUG [Thread-2192] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:41,225 DEBUG [Thread-2188] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:62055 2023-04-20 20:55:41,225 DEBUG [Thread-2188] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:41,227 DEBUG [Thread-2196] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eb8f142 to 127.0.0.1:62055 2023-04-20 20:55:41,227 DEBUG [Thread-2196] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:41,229 DEBUG [Thread-2194] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:62055 2023-04-20 20:55:41,229 DEBUG [Thread-2194] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:41,231 DEBUG [Thread-2190] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:62055 2023-04-20 20:55:41,231 DEBUG [Thread-2190] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:41,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024201284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024201285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024201286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,292 DEBUG [Thread-2187] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:41,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:41,366 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=724, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/2ac39bc2758c4d65ae9fdcc543b8fd21 2023-04-20 20:55:41,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024201587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024201587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:41,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024201588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:41,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/58c467f83f04443abde5fa09fba2f13c 2023-04-20 20:55:41,779 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-20 20:55:41,816 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/43f6a14ceb4047a1a65e58cfcdc2d491, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/afe9db1575064e8883510fc357ecf47b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6ded95415f1a442db0c999a9a1af5df9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9d0beb4c7725437ab88d2c7e07903024, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/4b5c34f679d745d8b91257e8df539db9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/c6975b878a4f401680dcde751f3e9f17, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5cee5c2446f14ef78de8b3f09b829fe9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/e6ba9d75e11e42958f3a8a3c99a10b3c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/da6c5a06357944d98843e9b9a0800534, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dec4f0d3bd60467d8c6a4d412631bc7b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/98df15fbc504450cae97cdd6221b02c1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/fc27ea171f4e4b53802950f21d0ec1d0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0bcd44da4904447cb5e5ab3e1f6d93c5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/622acff34b984d3a92a17a93a89ac0d1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/862e2f606fb049ea9d6803ae567af72b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/766bfe21275f43f4b4cc0ed8c7b2df65, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5379e2213f18481b85d3f692af5ec1d2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/398aff1c288e4311b851be1f23772508, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/905053e5ab9947839249a33945f6b985, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9518b65d187744e380990ea5906db3c7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6707072e9f2c4835b0c14f494ec060b3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9f91c35b5ac94621b576402055f2a44f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dd5726e63c0f4b5fb0fef382523b51b9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/b382eb49c56746889a6a7d5128eb6646, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/822287b7620646a69decbe98128f619f] to archive 2023-04-20 20:55:41,816 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:41,818 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/43f6a14ceb4047a1a65e58cfcdc2d491 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/43f6a14ceb4047a1a65e58cfcdc2d491 2023-04-20 20:55:41,819 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/afe9db1575064e8883510fc357ecf47b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/afe9db1575064e8883510fc357ecf47b 2023-04-20 20:55:41,820 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6ded95415f1a442db0c999a9a1af5df9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6ded95415f1a442db0c999a9a1af5df9 2023-04-20 20:55:41,820 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9d0beb4c7725437ab88d2c7e07903024 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9d0beb4c7725437ab88d2c7e07903024 2023-04-20 20:55:41,821 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/4b5c34f679d745d8b91257e8df539db9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/4b5c34f679d745d8b91257e8df539db9 2023-04-20 20:55:41,822 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/c6975b878a4f401680dcde751f3e9f17 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/c6975b878a4f401680dcde751f3e9f17 2023-04-20 20:55:41,823 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5cee5c2446f14ef78de8b3f09b829fe9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5cee5c2446f14ef78de8b3f09b829fe9 2023-04-20 20:55:41,824 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/e6ba9d75e11e42958f3a8a3c99a10b3c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/e6ba9d75e11e42958f3a8a3c99a10b3c 2023-04-20 20:55:41,824 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/da6c5a06357944d98843e9b9a0800534 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/da6c5a06357944d98843e9b9a0800534 2023-04-20 20:55:41,825 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dec4f0d3bd60467d8c6a4d412631bc7b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dec4f0d3bd60467d8c6a4d412631bc7b 2023-04-20 20:55:41,826 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/98df15fbc504450cae97cdd6221b02c1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/98df15fbc504450cae97cdd6221b02c1 2023-04-20 20:55:41,827 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/fc27ea171f4e4b53802950f21d0ec1d0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/fc27ea171f4e4b53802950f21d0ec1d0 2023-04-20 20:55:41,827 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0bcd44da4904447cb5e5ab3e1f6d93c5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0bcd44da4904447cb5e5ab3e1f6d93c5 2023-04-20 20:55:41,828 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/622acff34b984d3a92a17a93a89ac0d1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/622acff34b984d3a92a17a93a89ac0d1 2023-04-20 20:55:41,829 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/862e2f606fb049ea9d6803ae567af72b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/862e2f606fb049ea9d6803ae567af72b 2023-04-20 20:55:41,830 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/766bfe21275f43f4b4cc0ed8c7b2df65 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/766bfe21275f43f4b4cc0ed8c7b2df65 2023-04-20 20:55:41,830 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5379e2213f18481b85d3f692af5ec1d2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5379e2213f18481b85d3f692af5ec1d2 2023-04-20 20:55:41,831 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/398aff1c288e4311b851be1f23772508 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/398aff1c288e4311b851be1f23772508 2023-04-20 20:55:41,832 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/905053e5ab9947839249a33945f6b985 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/905053e5ab9947839249a33945f6b985 2023-04-20 20:55:41,833 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9518b65d187744e380990ea5906db3c7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9518b65d187744e380990ea5906db3c7 2023-04-20 20:55:41,834 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6707072e9f2c4835b0c14f494ec060b3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/6707072e9f2c4835b0c14f494ec060b3 2023-04-20 20:55:41,835 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9f91c35b5ac94621b576402055f2a44f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9f91c35b5ac94621b576402055f2a44f 2023-04-20 20:55:41,835 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dd5726e63c0f4b5fb0fef382523b51b9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/dd5726e63c0f4b5fb0fef382523b51b9 2023-04-20 20:55:41,836 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/b382eb49c56746889a6a7d5128eb6646 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/b382eb49c56746889a6a7d5128eb6646 2023-04-20 20:55:41,837 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/5d1bd5c4e96c4ce9b57534bf2d3c1d4d 2023-04-20 20:55:41,838 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/822287b7620646a69decbe98128f619f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/822287b7620646a69decbe98128f619f 2023-04-20 20:55:41,843 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0b51b2b02a1d42539ee501fdadd18599, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2a86c048be474a5eb11544abeb7341d2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/07b9bd9d07dc4afd9c3bb722db0dc644, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a005429fb1949c4a879ce6272ffeb03, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/9fef4a1c02eb4e70b4d4ea1f03310385, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a9a63b0dcd44ecd979169b3a538a8b7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/91184d7b3a584cc291e848befefd7720, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/266d02320bde4fa28bd6c78d09da0137, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/d3f1e051df484ae0871f3874d97a14e0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a1662b221cf643f8aab1413e27e084e0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/1e78124cd638450cb11a1b3ecdc82379, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/52685ee295134f66a83019c11e827ae0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0405c64a28d740f6bfe593698628672b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a158895da6d94024a86e2032615b397a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2f6912366c8e472f9e4cc51595bb4125, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/025c149e6d3845bb8f156d6c982d4221, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/be789d1bea9846449c6eb0faf8ae85db, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/fec386242957411b8502ee5f9526ac5d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/dd64747d220649f6bacedd2911b08f69, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3fd5060d50894bf48c35fd3033e1b4d5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/684e759f4bd946898253017dc0ae5078, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/b5d15e4103514444b64bb745d7e8fb4d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/4adb9ea075914323872eaaee4e09c724, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/e3d7b0d64be142d4ba1540700c212328, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3736ee2195794f038581c953252c19c5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/584759f3fc484a3e864fe819e4d712db] to archive 2023-04-20 20:55:41,844 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:41,845 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0b51b2b02a1d42539ee501fdadd18599 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0b51b2b02a1d42539ee501fdadd18599 2023-04-20 20:55:41,846 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2a86c048be474a5eb11544abeb7341d2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2a86c048be474a5eb11544abeb7341d2 2023-04-20 20:55:41,846 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/07b9bd9d07dc4afd9c3bb722db0dc644 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/07b9bd9d07dc4afd9c3bb722db0dc644 2023-04-20 20:55:41,847 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a005429fb1949c4a879ce6272ffeb03 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a005429fb1949c4a879ce6272ffeb03 2023-04-20 20:55:41,848 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/9fef4a1c02eb4e70b4d4ea1f03310385 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/9fef4a1c02eb4e70b4d4ea1f03310385 2023-04-20 20:55:41,849 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a9a63b0dcd44ecd979169b3a538a8b7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/7a9a63b0dcd44ecd979169b3a538a8b7 2023-04-20 20:55:41,850 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/91184d7b3a584cc291e848befefd7720 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/91184d7b3a584cc291e848befefd7720 2023-04-20 20:55:41,850 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/266d02320bde4fa28bd6c78d09da0137 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/266d02320bde4fa28bd6c78d09da0137 2023-04-20 20:55:41,851 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/d3f1e051df484ae0871f3874d97a14e0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/d3f1e051df484ae0871f3874d97a14e0 2023-04-20 20:55:41,852 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a1662b221cf643f8aab1413e27e084e0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a1662b221cf643f8aab1413e27e084e0 2023-04-20 20:55:41,853 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/1e78124cd638450cb11a1b3ecdc82379 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/1e78124cd638450cb11a1b3ecdc82379 2023-04-20 20:55:41,853 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/52685ee295134f66a83019c11e827ae0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/52685ee295134f66a83019c11e827ae0 2023-04-20 20:55:41,854 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0405c64a28d740f6bfe593698628672b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/0405c64a28d740f6bfe593698628672b 2023-04-20 20:55:41,855 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a158895da6d94024a86e2032615b397a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/a158895da6d94024a86e2032615b397a 2023-04-20 20:55:41,856 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2f6912366c8e472f9e4cc51595bb4125 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/2f6912366c8e472f9e4cc51595bb4125 2023-04-20 20:55:41,857 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/025c149e6d3845bb8f156d6c982d4221 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/025c149e6d3845bb8f156d6c982d4221 2023-04-20 20:55:41,857 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/be789d1bea9846449c6eb0faf8ae85db to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/be789d1bea9846449c6eb0faf8ae85db 2023-04-20 20:55:41,858 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/fec386242957411b8502ee5f9526ac5d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/fec386242957411b8502ee5f9526ac5d 2023-04-20 20:55:41,859 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/dd64747d220649f6bacedd2911b08f69 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/dd64747d220649f6bacedd2911b08f69 2023-04-20 20:55:41,859 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3fd5060d50894bf48c35fd3033e1b4d5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3fd5060d50894bf48c35fd3033e1b4d5 2023-04-20 20:55:41,860 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/684e759f4bd946898253017dc0ae5078 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/684e759f4bd946898253017dc0ae5078 2023-04-20 20:55:41,861 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/b5d15e4103514444b64bb745d7e8fb4d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/b5d15e4103514444b64bb745d7e8fb4d 2023-04-20 20:55:41,861 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/4adb9ea075914323872eaaee4e09c724 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/4adb9ea075914323872eaaee4e09c724 2023-04-20 20:55:41,862 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/e3d7b0d64be142d4ba1540700c212328 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/e3d7b0d64be142d4ba1540700c212328 2023-04-20 20:55:41,863 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3736ee2195794f038581c953252c19c5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/3736ee2195794f038581c953252c19c5 2023-04-20 20:55:41,863 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/584759f3fc484a3e864fe819e4d712db to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/584759f3fc484a3e864fe819e4d712db 2023-04-20 20:55:41,868 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/b52e91fec7154df6aec60019bd0532c2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/05f2dfb8a1c44eb293ae7e9230c9f9aa, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a085f0ff2a084ea4a15898f0903f4f51, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/36f9b307168c4255a98c06f6ab8334e7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1773933967db4efcbe8f3bf90db89a8b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/01dc8d74916b4d63a1fdedd23456c8b2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0526f15c1f4445e19dee2b44031f82bf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f7d99880114549e5a8ef732e3e638f76, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/69f2124ebca249a79108ce4297bf2f2e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/4300a471364c4990a5d629fe2ead5bc8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e53dbd24349c489692ac20f2e272799b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/68cc14a4e38f4a83af2f641c119869b6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/5c67c05ef3444465b2b9d835b6a16218, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f107ea01716b4e65ae7dedd00d6d295e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/113cdf5d3a014f36afa9a7b4905fd796, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1e975e73c1ad443db303f03148902ab5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/297746e8ae354be183b8105706ddf551, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e6ab74c3f6274b6aa386c1408ec44e06, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/c6acb3df72374e81a828391d20cec8fe, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a5e8236dec584c779316346381708de6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e5d5df441a9a4ce385c12844ed88b580, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0bcb32adefd248b0bc5ab9c0a7de4ef7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/7b30b2a360ab4c4984c66cb90e99a097, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e79686ffcdc04a0eb999772553557ed1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0b233e19ad404b47a372a7a9b5c005bf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/975d0a73e4c1435f938d5f7b37d83dd4] to archive 2023-04-20 20:55:41,868 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:41,869 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/b52e91fec7154df6aec60019bd0532c2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/b52e91fec7154df6aec60019bd0532c2 2023-04-20 20:55:41,870 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/05f2dfb8a1c44eb293ae7e9230c9f9aa to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/05f2dfb8a1c44eb293ae7e9230c9f9aa 2023-04-20 20:55:41,871 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a085f0ff2a084ea4a15898f0903f4f51 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a085f0ff2a084ea4a15898f0903f4f51 2023-04-20 20:55:41,872 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/36f9b307168c4255a98c06f6ab8334e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/36f9b307168c4255a98c06f6ab8334e7 2023-04-20 20:55:41,872 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1773933967db4efcbe8f3bf90db89a8b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1773933967db4efcbe8f3bf90db89a8b 2023-04-20 20:55:41,873 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/01dc8d74916b4d63a1fdedd23456c8b2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/01dc8d74916b4d63a1fdedd23456c8b2 2023-04-20 20:55:41,874 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0526f15c1f4445e19dee2b44031f82bf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0526f15c1f4445e19dee2b44031f82bf 2023-04-20 20:55:41,875 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f7d99880114549e5a8ef732e3e638f76 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f7d99880114549e5a8ef732e3e638f76 2023-04-20 20:55:41,875 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/69f2124ebca249a79108ce4297bf2f2e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/69f2124ebca249a79108ce4297bf2f2e 2023-04-20 20:55:41,876 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/4300a471364c4990a5d629fe2ead5bc8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/4300a471364c4990a5d629fe2ead5bc8 2023-04-20 20:55:41,877 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e53dbd24349c489692ac20f2e272799b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e53dbd24349c489692ac20f2e272799b 2023-04-20 20:55:41,878 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/68cc14a4e38f4a83af2f641c119869b6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/68cc14a4e38f4a83af2f641c119869b6 2023-04-20 20:55:41,878 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/5c67c05ef3444465b2b9d835b6a16218 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/5c67c05ef3444465b2b9d835b6a16218 2023-04-20 20:55:41,879 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f107ea01716b4e65ae7dedd00d6d295e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/f107ea01716b4e65ae7dedd00d6d295e 2023-04-20 20:55:41,880 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/113cdf5d3a014f36afa9a7b4905fd796 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/113cdf5d3a014f36afa9a7b4905fd796 2023-04-20 20:55:41,881 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1e975e73c1ad443db303f03148902ab5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/1e975e73c1ad443db303f03148902ab5 2023-04-20 20:55:41,881 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/297746e8ae354be183b8105706ddf551 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/297746e8ae354be183b8105706ddf551 2023-04-20 20:55:41,882 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e6ab74c3f6274b6aa386c1408ec44e06 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e6ab74c3f6274b6aa386c1408ec44e06 2023-04-20 20:55:41,883 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/c6acb3df72374e81a828391d20cec8fe to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/c6acb3df72374e81a828391d20cec8fe 2023-04-20 20:55:41,884 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a5e8236dec584c779316346381708de6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a5e8236dec584c779316346381708de6 2023-04-20 20:55:41,885 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e5d5df441a9a4ce385c12844ed88b580 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e5d5df441a9a4ce385c12844ed88b580 2023-04-20 20:55:41,885 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0bcb32adefd248b0bc5ab9c0a7de4ef7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0bcb32adefd248b0bc5ab9c0a7de4ef7 2023-04-20 20:55:41,886 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/7b30b2a360ab4c4984c66cb90e99a097 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/7b30b2a360ab4c4984c66cb90e99a097 2023-04-20 20:55:41,887 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e79686ffcdc04a0eb999772553557ed1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/e79686ffcdc04a0eb999772553557ed1 2023-04-20 20:55:41,888 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0b233e19ad404b47a372a7a9b5c005bf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/0b233e19ad404b47a372a7a9b5c005bf 2023-04-20 20:55:41,888 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/975d0a73e4c1435f938d5f7b37d83dd4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/975d0a73e4c1435f938d5f7b37d83dd4 2023-04-20 20:55:42,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44534 deadline: 1682024202088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:42,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:42,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44600 deadline: 1682024202089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44568 deadline: 1682024202089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:42,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/701768af1b2a406e94df349c210ce471 2023-04-20 20:55:42,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/2ac39bc2758c4d65ae9fdcc543b8fd21 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2ac39bc2758c4d65ae9fdcc543b8fd21 2023-04-20 20:55:42,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2ac39bc2758c4d65ae9fdcc543b8fd21, entries=200, sequenceid=724, filesize=32.3 K 2023-04-20 20:55:42,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/58c467f83f04443abde5fa09fba2f13c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/58c467f83f04443abde5fa09fba2f13c 2023-04-20 20:55:42,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/58c467f83f04443abde5fa09fba2f13c, entries=150, sequenceid=724, filesize=11.9 K 2023-04-20 20:55:42,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/701768af1b2a406e94df349c210ce471 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/701768af1b2a406e94df349c210ce471 2023-04-20 20:55:42,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/701768af1b2a406e94df349c210ce471, entries=150, sequenceid=724, filesize=11.9 K 2023-04-20 20:55:42,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1296ms, sequenceid=724, compaction requested=true 2023-04-20 20:55:42,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:42,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:42,232 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:42,232 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:42,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:42,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:42,233 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86064 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:42,233 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:42,233 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/A is initiating minor compaction (all files) 2023-04-20 20:55:42,234 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/B is initiating minor compaction (all files) 2023-04-20 20:55:42,234 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/A in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:42,234 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/B in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:42,234 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/640498cabadc40848ec7883a2787ff76, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9ad4b4fe7fb346e7a84606de7d68c21d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2ac39bc2758c4d65ae9fdcc543b8fd21] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=84.0 K 2023-04-20 20:55:42,234 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f727de012faa42128a308bc7c2f2116c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f3573ae1cd4e4376904e39f3a9fb2437, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/58c467f83f04443abde5fa09fba2f13c] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=36.6 K 2023-04-20 20:55:42,234 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 640498cabadc40848ec7883a2787ff76, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682024137786 2023-04-20 20:55:42,234 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f727de012faa42128a308bc7c2f2116c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682024137786 2023-04-20 20:55:42,234 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9ad4b4fe7fb346e7a84606de7d68c21d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1682024138147 2023-04-20 20:55:42,234 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f3573ae1cd4e4376904e39f3a9fb2437, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1682024138147 2023-04-20 20:55:42,235 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2ac39bc2758c4d65ae9fdcc543b8fd21, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1682024140305 2023-04-20 20:55:42,235 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 58c467f83f04443abde5fa09fba2f13c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1682024140305 2023-04-20 20:55:42,240 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#B#compaction#378 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:42,245 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#A#compaction#377 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:42,653 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/857f6423ddd14d0e80e76e7527168ead as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/857f6423ddd14d0e80e76e7527168ead 2023-04-20 20:55:42,658 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/B of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 857f6423ddd14d0e80e76e7527168ead(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:42,658 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:42,658 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/B, priority=13, startTime=1682024142232; duration=0sec 2023-04-20 20:55:42,658 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:42,658 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:42,659 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:42,659 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 7dcdfe0ddb3c517a4a0d9b1a81351709/C is initiating minor compaction (all files) 2023-04-20 20:55:42,659 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7dcdfe0ddb3c517a4a0d9b1a81351709/C in TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:42,659 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/cac6516794eb4e8490543c70e446ea9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/82fd61c99763425998693540bf1d7121, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/701768af1b2a406e94df349c210ce471] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp, totalSize=36.6 K 2023-04-20 20:55:42,660 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting cac6516794eb4e8490543c70e446ea9d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1682024137786 2023-04-20 20:55:42,660 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 82fd61c99763425998693540bf1d7121, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1682024138147 2023-04-20 20:55:42,660 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 701768af1b2a406e94df349c210ce471, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1682024140305 2023-04-20 20:55:42,664 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 7dcdfe0ddb3c517a4a0d9b1a81351709#C#compaction#379 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:43,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:43,010 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 20:55:43,010 DEBUG [Thread-2179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:62055 2023-04-20 20:55:43,010 DEBUG [Thread-2179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:43,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:43,011 DEBUG [Thread-2177] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:62055 2023-04-20 20:55:43,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:43,011 DEBUG [Thread-2177] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:43,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:43,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:43,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:43,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:43,072 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/da505e158c6a47ad8cfc0792282bf0e3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/da505e158c6a47ad8cfc0792282bf0e3 2023-04-20 20:55:43,078 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/C of 7dcdfe0ddb3c517a4a0d9b1a81351709 into da505e158c6a47ad8cfc0792282bf0e3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:43,078 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:43,078 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/C, priority=13, startTime=1682024142233; duration=0sec 2023-04-20 20:55:43,078 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:43,091 DEBUG [Thread-2183] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:62055 2023-04-20 20:55:43,091 DEBUG [Thread-2183] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:43,097 DEBUG [Thread-2185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:62055 2023-04-20 20:55:43,097 DEBUG [Thread-2185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:43,098 DEBUG [Thread-2181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:62055 2023-04-20 20:55:43,098 DEBUG [Thread-2181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:43,098 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 187 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1620 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 4852 rows 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1614 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 4842 rows 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1617 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 4849 rows 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1618 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 4852 rows 2023-04-20 20:55:43,099 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1618 2023-04-20 20:55:43,100 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 4854 rows 2023-04-20 20:55:43,100 INFO [Listener at localhost/45619] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 20:55:43,100 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:62055 2023-04-20 20:55:43,100 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:55:43,103 INFO [Listener at localhost/45619] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 20:55:43,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-20 20:55:43,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:43,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 20:55:43,109 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024143109"}]},"ts":"1682024143109"} 2023-04-20 20:55:43,110 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 20:55:43,112 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 20:55:43,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, UNASSIGN}] 2023-04-20 20:55:43,115 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, UNASSIGN 2023-04-20 20:55:43,116 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:43,116 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024143116"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024143116"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024143116"}]},"ts":"1682024143116"} 2023-04-20 20:55:43,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:55:43,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 20:55:43,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:43,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7dcdfe0ddb3c517a4a0d9b1a81351709, disabling compactions & flushes 2023-04-20 20:55:43,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:43,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 20:55:43,421 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042068de2971220145debfa7a4f2ec9e4d88 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042068de2971220145debfa7a4f2ec9e4d88 2023-04-20 20:55:43,466 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/0c9f293389bc475bb39c5be9df44743d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0c9f293389bc475bb39c5be9df44743d 2023-04-20 20:55:43,469 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7dcdfe0ddb3c517a4a0d9b1a81351709/A of 7dcdfe0ddb3c517a4a0d9b1a81351709 into 0c9f293389bc475bb39c5be9df44743d(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-20 20:55:43,469 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:43,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:43,469 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709., storeName=7dcdfe0ddb3c517a4a0d9b1a81351709/A, priority=13, startTime=1682024142232; duration=1sec 2023-04-20 20:55:43,469 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:43,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 20:55:43,825 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=744, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/2aaac38079764ea99966a25a96207440 2023-04-20 20:55:44,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 20:55:44,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/875243b55ab74c63baf5c7fca2660876 2023-04-20 20:55:44,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/a2d59f4d752447e7b804c410e16a44d1 2023-04-20 20:55:44,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/2aaac38079764ea99966a25a96207440 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2aaac38079764ea99966a25a96207440 2023-04-20 20:55:44,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2aaac38079764ea99966a25a96207440, entries=150, sequenceid=744, filesize=25.4 K 2023-04-20 20:55:44,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/875243b55ab74c63baf5c7fca2660876 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/875243b55ab74c63baf5c7fca2660876 2023-04-20 20:55:44,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/875243b55ab74c63baf5c7fca2660876, entries=150, sequenceid=744, filesize=11.9 K 2023-04-20 20:55:44,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/a2d59f4d752447e7b804c410e16a44d1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a2d59f4d752447e7b804c410e16a44d1 2023-04-20 20:55:44,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a2d59f4d752447e7b804c410e16a44d1, entries=150, sequenceid=744, filesize=11.9 K 2023-04-20 20:55:44,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=20.13 KB/20610 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1649ms, sequenceid=744, compaction requested=false 2023-04-20 20:55:44,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:44,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:44,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:44,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. after waiting 0 ms 2023-04-20 20:55:44,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:44,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7dcdfe0ddb3c517a4a0d9b1a81351709 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-20 20:55:44,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=A 2023-04-20 20:55:44,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:44,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=B 2023-04-20 20:55:44,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:44,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7dcdfe0ddb3c517a4a0d9b1a81351709, store=C 2023-04-20 20:55:44,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:45,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304205c0358f6e2704b3cb8701b230915480d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205c0358f6e2704b3cb8701b230915480d 2023-04-20 20:55:45,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 20:55:45,476 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=752, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/ba3a95952dbb4b3a8a9e1398329efde1 2023-04-20 20:55:45,886 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/5b531670b2124eb99a0d3bf2bc9621e0 2023-04-20 20:55:46,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/ab0d522765e946949372ced04dddd511 2023-04-20 20:55:46,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/A/ba3a95952dbb4b3a8a9e1398329efde1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/ba3a95952dbb4b3a8a9e1398329efde1 2023-04-20 20:55:46,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/ba3a95952dbb4b3a8a9e1398329efde1, entries=100, sequenceid=752, filesize=18.6 K 2023-04-20 20:55:46,303 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/B/5b531670b2124eb99a0d3bf2bc9621e0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/5b531670b2124eb99a0d3bf2bc9621e0 2023-04-20 20:55:46,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/5b531670b2124eb99a0d3bf2bc9621e0, entries=100, sequenceid=752, filesize=9.5 K 2023-04-20 20:55:46,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/.tmp/C/ab0d522765e946949372ced04dddd511 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/ab0d522765e946949372ced04dddd511 2023-04-20 20:55:46,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/ab0d522765e946949372ced04dddd511, entries=100, sequenceid=752, filesize=9.5 K 2023-04-20 20:55:46,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 7dcdfe0ddb3c517a4a0d9b1a81351709 in 1652ms, sequenceid=752, compaction requested=true 2023-04-20 20:55:46,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/640498cabadc40848ec7883a2787ff76, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9ad4b4fe7fb346e7a84606de7d68c21d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2ac39bc2758c4d65ae9fdcc543b8fd21] to archive 2023-04-20 20:55:46,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:46,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/640498cabadc40848ec7883a2787ff76 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/640498cabadc40848ec7883a2787ff76 2023-04-20 20:55:46,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9ad4b4fe7fb346e7a84606de7d68c21d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/9ad4b4fe7fb346e7a84606de7d68c21d 2023-04-20 20:55:46,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2ac39bc2758c4d65ae9fdcc543b8fd21 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2ac39bc2758c4d65ae9fdcc543b8fd21 2023-04-20 20:55:46,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f727de012faa42128a308bc7c2f2116c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f3573ae1cd4e4376904e39f3a9fb2437, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/58c467f83f04443abde5fa09fba2f13c] to archive 2023-04-20 20:55:46,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:46,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f727de012faa42128a308bc7c2f2116c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f727de012faa42128a308bc7c2f2116c 2023-04-20 20:55:46,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f3573ae1cd4e4376904e39f3a9fb2437 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/f3573ae1cd4e4376904e39f3a9fb2437 2023-04-20 20:55:46,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/58c467f83f04443abde5fa09fba2f13c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/58c467f83f04443abde5fa09fba2f13c 2023-04-20 20:55:46,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/cac6516794eb4e8490543c70e446ea9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/82fd61c99763425998693540bf1d7121, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/701768af1b2a406e94df349c210ce471] to archive 2023-04-20 20:55:46,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:55:46,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/cac6516794eb4e8490543c70e446ea9d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/cac6516794eb4e8490543c70e446ea9d 2023-04-20 20:55:46,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/82fd61c99763425998693540bf1d7121 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/82fd61c99763425998693540bf1d7121 2023-04-20 20:55:46,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/701768af1b2a406e94df349c210ce471 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/701768af1b2a406e94df349c210ce471 2023-04-20 20:55:46,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/recovered.edits/755.seqid, newMaxSeqId=755, maxSeqId=4 2023-04-20 20:55:46,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709. 2023-04-20 20:55:46,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7dcdfe0ddb3c517a4a0d9b1a81351709: 2023-04-20 20:55:46,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:46,331 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7dcdfe0ddb3c517a4a0d9b1a81351709, regionState=CLOSED 2023-04-20 20:55:46,331 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024146331"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024146331"}]},"ts":"1682024146331"} 2023-04-20 20:55:46,333 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-20 20:55:46,333 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 7dcdfe0ddb3c517a4a0d9b1a81351709, server=jenkins-hbase4.apache.org,39223,1682024020002 in 3.2150 sec 2023-04-20 20:55:46,334 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-20 20:55:46,334 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7dcdfe0ddb3c517a4a0d9b1a81351709, UNASSIGN in 3.2210 sec 2023-04-20 20:55:46,335 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024146335"}]},"ts":"1682024146335"} 2023-04-20 20:55:46,336 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 20:55:46,337 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 20:55:46,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.2340 sec 2023-04-20 20:55:47,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 20:55:47,214 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-20 20:55:47,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-20 20:55:47,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:47,217 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-20 20:55:47,218 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:47,222 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:47,224 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/recovered.edits] 2023-04-20 20:55:47,226 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0c9f293389bc475bb39c5be9df44743d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/0c9f293389bc475bb39c5be9df44743d 2023-04-20 20:55:47,227 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2aaac38079764ea99966a25a96207440 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/2aaac38079764ea99966a25a96207440 2023-04-20 20:55:47,228 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/ba3a95952dbb4b3a8a9e1398329efde1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/A/ba3a95952dbb4b3a8a9e1398329efde1 2023-04-20 20:55:47,231 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/5b531670b2124eb99a0d3bf2bc9621e0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/5b531670b2124eb99a0d3bf2bc9621e0 2023-04-20 20:55:47,232 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/857f6423ddd14d0e80e76e7527168ead to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/857f6423ddd14d0e80e76e7527168ead 2023-04-20 20:55:47,234 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/875243b55ab74c63baf5c7fca2660876 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/B/875243b55ab74c63baf5c7fca2660876 2023-04-20 20:55:47,235 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a2d59f4d752447e7b804c410e16a44d1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/a2d59f4d752447e7b804c410e16a44d1 2023-04-20 20:55:47,236 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/ab0d522765e946949372ced04dddd511 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/ab0d522765e946949372ced04dddd511 2023-04-20 20:55:47,237 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/da505e158c6a47ad8cfc0792282bf0e3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/C/da505e158c6a47ad8cfc0792282bf0e3 2023-04-20 20:55:47,240 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/recovered.edits/755.seqid to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709/recovered.edits/755.seqid 2023-04-20 20:55:47,240 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/7dcdfe0ddb3c517a4a0d9b1a81351709 2023-04-20 20:55:47,240 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:55:47,241 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 20:55:47,241 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-20 20:55:47,244 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200b2b65d4e357471986d7cb19c8403ae7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200b2b65d4e357471986d7cb19c8403ae7 2023-04-20 20:55:47,245 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420157b11c0e9ec454fb2e558e4169a0671 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420157b11c0e9ec454fb2e558e4169a0671 2023-04-20 20:55:47,246 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042022017daf7fbd441195ecb12d587c0fe4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042022017daf7fbd441195ecb12d587c0fe4 2023-04-20 20:55:47,247 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203cf142af191440639c825af0e6fbad91 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203cf142af191440639c825af0e6fbad91 2023-04-20 20:55:47,248 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042054690f29d76f493f9dae187cae4bac73 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042054690f29d76f493f9dae187cae4bac73 2023-04-20 20:55:47,249 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205b9db0f63df74ea8a41038a6c1f90f5f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205b9db0f63df74ea8a41038a6c1f90f5f 2023-04-20 20:55:47,250 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205c0358f6e2704b3cb8701b230915480d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205c0358f6e2704b3cb8701b230915480d 2023-04-20 20:55:47,250 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042068de2971220145debfa7a4f2ec9e4d88 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042068de2971220145debfa7a4f2ec9e4d88 2023-04-20 20:55:47,251 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206a59f3d71a304a129e0cf824c96dbcbd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206a59f3d71a304a129e0cf824c96dbcbd 2023-04-20 20:55:47,252 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420730deb53b2dd4a3293f2d0a72d9814e7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420730deb53b2dd4a3293f2d0a72d9814e7 2023-04-20 20:55:47,253 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207de3eb84cb8c40bd8a4ef5ef1f1d609d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207de3eb84cb8c40bd8a4ef5ef1f1d609d 2023-04-20 20:55:47,254 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208500b0ea26d44848ac894d6e50ebc94a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208500b0ea26d44848ac894d6e50ebc94a 2023-04-20 20:55:47,255 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208dd371eb15f648a390c0bb47d3704155 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208dd371eb15f648a390c0bb47d3704155 2023-04-20 20:55:47,256 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209a5cb414a7c44c22927b67669a82be70 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209a5cb414a7c44c22927b67669a82be70 2023-04-20 20:55:47,257 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b261d98ee1d84ce5bb75fa6d5b0d59e1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b261d98ee1d84ce5bb75fa6d5b0d59e1 2023-04-20 20:55:47,258 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420cf5049562fe44df18c5c32b9d440684a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420cf5049562fe44df18c5c32b9d440684a 2023-04-20 20:55:47,259 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d3775a7cf855449496d85847ef9ed1b1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d3775a7cf855449496d85847ef9ed1b1 2023-04-20 20:55:47,260 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e2a57395aebe44f888385dec69cc3d92 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e2a57395aebe44f888385dec69cc3d92 2023-04-20 20:55:47,261 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e465f15424c0457db4eaf917188c6bb6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e465f15424c0457db4eaf917188c6bb6 2023-04-20 20:55:47,262 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e73ea9943e1443efbd7fda9a9a0b25d5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e73ea9943e1443efbd7fda9a9a0b25d5 2023-04-20 20:55:47,263 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e963a736811e4cb697cf74ff80de4eb4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e963a736811e4cb697cf74ff80de4eb4 2023-04-20 20:55:47,264 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ed69c7de08d9463e80f55d347daa8e68 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ed69c7de08d9463e80f55d347daa8e68 2023-04-20 20:55:47,265 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fbed9e9d20ed4068bf59f8181dbfef9a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fbed9e9d20ed4068bf59f8181dbfef9a 2023-04-20 20:55:47,266 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fe55254002b2465392d8f636df5b2fc3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fe55254002b2465392d8f636df5b2fc3 2023-04-20 20:55:47,267 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 20:55:47,269 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:47,270 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 20:55:47,272 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 20:55:47,273 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:47,273 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 20:55:47,273 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682024147273"}]},"ts":"9223372036854775807"} 2023-04-20 20:55:47,274 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 20:55:47,274 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7dcdfe0ddb3c517a4a0d9b1a81351709, NAME => 'TestAcidGuarantees,,1682024119104.7dcdfe0ddb3c517a4a0d9b1a81351709.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 20:55:47,274 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 20:55:47,274 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682024147274"}]},"ts":"9223372036854775807"} 2023-04-20 20:55:47,275 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 20:55:47,277 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:47,278 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 62 msec 2023-04-20 20:55:47,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-20 20:55:47,319 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-20 20:55:47,337 INFO [Listener at localhost/45619] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=233 (was 227) - Thread LEAK? -, OpenFileDescriptor=551 (was 562), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 354) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=9184 (was 8657) - AvailableMemoryMB LEAK? - 2023-04-20 20:55:47,351 INFO [Listener at localhost/45619] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=233, OpenFileDescriptor=551, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=169, AvailableMemoryMB=9184 2023-04-20 20:55:47,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:55:47,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:55:47,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 20:55:47,357 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 20:55:47,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-20 20:55:47,358 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 20:55:47,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 20:55:47,359 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:47,359 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd empty. 2023-04-20 20:55:47,360 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:47,360 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:55:47,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 20:55:47,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 20:55:47,768 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:55:47,770 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8552cf5289bf62dd18e69f1f78b0b5fd, NAME => 'TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp 2023-04-20 20:55:47,786 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:55:47,786 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8552cf5289bf62dd18e69f1f78b0b5fd, disabling compactions & flushes 2023-04-20 20:55:47,786 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:47,787 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:47,787 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. after waiting 0 ms 2023-04-20 20:55:47,787 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:47,787 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:47,787 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:47,789 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 20:55:47,789 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024147789"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024147789"}]},"ts":"1682024147789"} 2023-04-20 20:55:47,790 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 20:55:47,791 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 20:55:47,791 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024147791"}]},"ts":"1682024147791"} 2023-04-20 20:55:47,792 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 20:55:47,795 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8552cf5289bf62dd18e69f1f78b0b5fd, ASSIGN}] 2023-04-20 20:55:47,797 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8552cf5289bf62dd18e69f1f78b0b5fd, ASSIGN 2023-04-20 20:55:47,798 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8552cf5289bf62dd18e69f1f78b0b5fd, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=false 2023-04-20 20:55:47,950 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=8552cf5289bf62dd18e69f1f78b0b5fd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:47,950 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024147950"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024147950"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024147950"}]},"ts":"1682024147950"} 2023-04-20 20:55:47,951 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:55:47,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 20:55:48,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8552cf5289bf62dd18e69f1f78b0b5fd, NAME => 'TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:55:48,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:55:48,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,108 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,109 DEBUG [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A 2023-04-20 20:55:48,109 DEBUG [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A 2023-04-20 20:55:48,110 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:48,110 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8552cf5289bf62dd18e69f1f78b0b5fd columnFamilyName A 2023-04-20 20:55:48,110 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.HStore(310): Store=8552cf5289bf62dd18e69f1f78b0b5fd/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:48,110 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,111 DEBUG [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B 2023-04-20 20:55:48,111 DEBUG [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B 2023-04-20 20:55:48,111 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:48,112 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8552cf5289bf62dd18e69f1f78b0b5fd columnFamilyName B 2023-04-20 20:55:48,112 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.HStore(310): Store=8552cf5289bf62dd18e69f1f78b0b5fd/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:48,112 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,113 DEBUG [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C 2023-04-20 20:55:48,113 DEBUG [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C 2023-04-20 20:55:48,113 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:55:48,113 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8552cf5289bf62dd18e69f1f78b0b5fd columnFamilyName C 2023-04-20 20:55:48,114 INFO [StoreOpener-8552cf5289bf62dd18e69f1f78b0b5fd-1] regionserver.HStore(310): Store=8552cf5289bf62dd18e69f1f78b0b5fd/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:55:48,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:55:48,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:55:48,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8552cf5289bf62dd18e69f1f78b0b5fd; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64952294, jitterRate=-0.03213539719581604}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:55:48,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:48,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., pid=49, masterSystemTime=1682024148103 2023-04-20 20:55:48,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,122 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=8552cf5289bf62dd18e69f1f78b0b5fd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,122 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024148122"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024148122"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024148122"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024148122"}]},"ts":"1682024148122"} 2023-04-20 20:55:48,124 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-20 20:55:48,124 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 in 172 msec 2023-04-20 20:55:48,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-20 20:55:48,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8552cf5289bf62dd18e69f1f78b0b5fd, ASSIGN in 329 msec 2023-04-20 20:55:48,126 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 20:55:48,126 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024148126"}]},"ts":"1682024148126"} 2023-04-20 20:55:48,127 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 20:55:48,129 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 20:55:48,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 775 msec 2023-04-20 20:55:48,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 20:55:48,463 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-20 20:55:48,464 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,469 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,471 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,473 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39980, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,474 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 20:55:48,476 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 20:55:48,477 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,482 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,482 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,492 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,493 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,498 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,498 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,503 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,504 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,511 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,512 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,520 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,521 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,530 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,531 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,539 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,539 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,547 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,547 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x4ac98310 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:55:48,555 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a9f3972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:55:48,564 DEBUG [hconnection-0x2be51aad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,564 DEBUG [hconnection-0x671b6391-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,565 DEBUG [hconnection-0x2bf3bc68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,566 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,571 DEBUG [hconnection-0x4f6d84b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,574 DEBUG [hconnection-0xfbbc2fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,574 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,574 DEBUG [hconnection-0xf2833cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,575 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,584 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:48,584 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39984, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,585 DEBUG [hconnection-0x45386018-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,585 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40020, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,586 DEBUG [hconnection-0x4f1fcc37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,586 DEBUG [hconnection-0x697c440e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,587 DEBUG [hconnection-0x5b606aa1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:55:48,590 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,590 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,602 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,602 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:55:48,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:48,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:48,607 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:48,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:48,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:48,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:48,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:55:48,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:48,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:48,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:48,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:48,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:48,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:48,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,627 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:48,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:48,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:48,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:48,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:48,631 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:48,632 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,637 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:48,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fbf6457[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:48,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:48,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:48,643 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,643 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:48,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/4d25dcef4cc34626956fad317658ddc0 2023-04-20 20:55:48,645 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:48,650 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:48,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:48,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:48,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:48,650 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:48,651 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,651 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:48,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,652 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024208641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024208643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024208651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024148636, max timestamp=1682024148652Num uniques 859; 2023-04-20 20:55:48,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024148636, max timestamp=1682024148652Num uniques 859; 2023-04-20 20:55:48,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024208654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:48,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:48,658 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:48,658 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:48,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:48,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024148636, max timestamp=1682024148652Num uniques 859; 2023-04-20 20:55:48,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:48,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:48,659 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,659 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:48,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:48,659 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:48,659 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:48,660 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:48,659 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:48,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024208700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024208755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024208755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024208757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,760 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:48,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024208758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:48,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:48,776 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:48,776 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:48,776 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:48,776 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:48,777 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,777 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,779 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:48,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:48,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:48,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:48,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:48,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:48,784 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:48,784 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:48,784 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:48,784 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:48,784 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:48,784 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,784 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:48,785 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:48,785 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:48,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:48,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:48,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:48,787 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,787 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,787 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,787 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16047112[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:48,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:48,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:48,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:48,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:48,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,791 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:48,792 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:48,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:48,796 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:48,796 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:48,796 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:48,796 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:48,796 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:48,798 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,799 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,799 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:48,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,801 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:48,804 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:48,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:48,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:48,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,804 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:48,804 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:48,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:48,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024208809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,905 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:48,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:48,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:48,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:48,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:48,912 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:48,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:48,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:48,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:48,913 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,913 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,915 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:48,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:48,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:48,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:48,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:48,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:48,920 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:48,920 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:48,920 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:48,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:48,924 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:48,924 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:48,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:48,925 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:48,934 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@caecaa5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:48,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:48,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,937 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:48,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:48,939 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:48,941 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:48,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:48,942 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:48,942 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:48,942 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:48,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:48,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,945 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:48,946 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:48,949 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:48,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:48,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:48,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:48,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:48,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:48,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:48,950 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:48,950 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:48,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:48,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024208960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024208961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024208963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:48,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:48,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024208967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024209015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,050 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:49,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:49,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:49,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:49,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:49,054 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:49,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:49,054 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:49,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,056 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:49,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:49,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:49,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:49,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:49,058 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:49,060 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,062 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,062 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3039a8ae[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:49,062 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:49,062 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,064 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:49,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:49,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:49,071 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:49,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:49,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:49,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:49,071 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:49,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/fb91de30ec6140f6874774a3c0911c78 2023-04-20 20:55:49,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:49,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:49,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,081 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:49,081 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:49,181 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:49,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:49,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:49,185 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:49,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,185 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:49,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:49,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,187 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,187 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:49,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:49,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:49,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,189 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,193 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1480ff22[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:49,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:49,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,196 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:49,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:49,198 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:49,203 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:49,203 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:49,203 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:49,203 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:49,203 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:49,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,214 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:49,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:49,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,215 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:49,215 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,215 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024209266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024209266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024209267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024209275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,315 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:49,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:49,319 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:49,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:49,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:49,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,321 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,321 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,321 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,321 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:49,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:49,323 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:49,323 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,323 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:49,323 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:49,323 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:49,325 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024209322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:49,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21398041[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:49,329 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:49,329 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:49,331 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,331 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:49,338 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:49,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:49,340 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:49,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:49,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:49,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:49,341 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:49,342 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,342 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,343 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,349 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:49,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:49,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,349 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:49,349 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:49,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:49,452 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:49,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:49,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:49,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:49,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:49,456 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:49,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:49,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:49,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,458 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,458 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:49,461 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,461 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:49,461 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:49,461 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:49,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:49,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:49,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:49,463 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,463 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,464 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb88d21[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:49,466 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:49,466 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:49,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:49,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,469 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:49,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:49,477 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:49,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:49,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:49,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:49,477 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:49,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:49,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:49,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,481 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:49,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:49,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:49,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:49,485 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:49,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:49,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:49,486 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:49,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/ef8d8897201145b099236b9265d9a28d 2023-04-20 20:55:49,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/4d25dcef4cc34626956fad317658ddc0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4d25dcef4cc34626956fad317658ddc0 2023-04-20 20:55:49,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4d25dcef4cc34626956fad317658ddc0, entries=350, sequenceid=13, filesize=20.8 K 2023-04-20 20:55:49,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/fb91de30ec6140f6874774a3c0911c78 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fb91de30ec6140f6874774a3c0911c78 2023-04-20 20:55:49,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fb91de30ec6140f6874774a3c0911c78, entries=150, sequenceid=13, filesize=11.6 K 2023-04-20 20:55:49,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/ef8d8897201145b099236b9265d9a28d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/ef8d8897201145b099236b9265d9a28d 2023-04-20 20:55:49,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/ef8d8897201145b099236b9265d9a28d, entries=150, sequenceid=13, filesize=11.6 K 2023-04-20 20:55:49,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.53 KB/121370 for 8552cf5289bf62dd18e69f1f78b0b5fd in 923ms, sequenceid=13, compaction requested=false 2023-04-20 20:55:49,531 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:49,586 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:49,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:49,590 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:49,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:49,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:49,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,592 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:49,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:49,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:49,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:49,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:49,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:49,608 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:49,608 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:49,608 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=118.53 KB heapSize=335.38 KB 2023-04-20 20:55:49,608 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:49,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:49,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:49,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:49,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:49,609 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:49,675 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/96a5c966f6924e34ba673019e1436082 2023-04-20 20:55:49,708 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/a7b07ffb4a034b4f9a31d1fa11f3413f 2023-04-20 20:55:49,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:49,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:49,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024209791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024209796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024209797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024209798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024209832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024209899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024209899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024209904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:49,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:49,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024209905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024210103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024210103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,121 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/70cc2a822968454780e3c3c53f18eef3 2023-04-20 20:55:50,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024210115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024210115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/96a5c966f6924e34ba673019e1436082 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/96a5c966f6924e34ba673019e1436082 2023-04-20 20:55:50,128 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/96a5c966f6924e34ba673019e1436082, entries=150, sequenceid=47, filesize=11.6 K 2023-04-20 20:55:50,129 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/a7b07ffb4a034b4f9a31d1fa11f3413f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a7b07ffb4a034b4f9a31d1fa11f3413f 2023-04-20 20:55:50,132 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a7b07ffb4a034b4f9a31d1fa11f3413f, entries=150, sequenceid=47, filesize=11.6 K 2023-04-20 20:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/70cc2a822968454780e3c3c53f18eef3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70cc2a822968454780e3c3c53f18eef3 2023-04-20 20:55:50,139 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70cc2a822968454780e3c3c53f18eef3, entries=150, sequenceid=47, filesize=11.6 K 2023-04-20 20:55:50,140 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.53 KB/121370, heapSize ~335.33 KB/343376, currentSize=67.09 KB/68700 for 8552cf5289bf62dd18e69f1f78b0b5fd in 532ms, sequenceid=47, compaction requested=false 2023-04-20 20:55:50,141 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:50,141 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:50,142 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b93220[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:50,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:50,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,148 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:50,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:50,148 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,148 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:50,153 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:50,159 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:50,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:50,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:50,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:50,159 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:50,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:50,161 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:50,163 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:50,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:50,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:50,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:50,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:50,166 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:50,167 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:50,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:50,267 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:50,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:50,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:50,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:50,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:50,272 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:50,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:50,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:50,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:50,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:50,274 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:50,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:50,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:50,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:50,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:50,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:50,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:50,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:50,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:50,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:50,276 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 20:55:50,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:50,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:50,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:50,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:50,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:50,277 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:50,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:50,418 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:50,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024210441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024210442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024210446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024210448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024210551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024210551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024210552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024210553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,690 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/a5ebe368ddf94ab99470bcd93bfa5f9d 2023-04-20 20:55:50,701 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/86df1595b92948b3ab966e47f2abfa4b 2023-04-20 20:55:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024210754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024210754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024210754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024210755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:50,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024210845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024211060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024211060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024211061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024211061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,114 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/a0ae507575974742918043a4ca5a68ad 2023-04-20 20:55:51,118 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/a5ebe368ddf94ab99470bcd93bfa5f9d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a5ebe368ddf94ab99470bcd93bfa5f9d 2023-04-20 20:55:51,121 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a5ebe368ddf94ab99470bcd93bfa5f9d, entries=150, sequenceid=60, filesize=11.6 K 2023-04-20 20:55:51,122 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/86df1595b92948b3ab966e47f2abfa4b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86df1595b92948b3ab966e47f2abfa4b 2023-04-20 20:55:51,126 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86df1595b92948b3ab966e47f2abfa4b, entries=150, sequenceid=60, filesize=11.6 K 2023-04-20 20:55:51,127 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/a0ae507575974742918043a4ca5a68ad as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a0ae507575974742918043a4ca5a68ad 2023-04-20 20:55:51,132 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a0ae507575974742918043a4ca5a68ad, entries=150, sequenceid=60, filesize=11.6 K 2023-04-20 20:55:51,133 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8552cf5289bf62dd18e69f1f78b0b5fd in 857ms, sequenceid=60, compaction requested=true 2023-04-20 20:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:51,133 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:51,133 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:51,136 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3553b03a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,139 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,139 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:51,141 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,141 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:51,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:51,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:51,142 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:51,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:51,145 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:51,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:51,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:51,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:51,145 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:51,150 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,150 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,150 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,151 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,152 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:51,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,156 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:51,156 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:51,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:51,257 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:51,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:51,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:51,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:51,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:51,260 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:51,260 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,260 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:51,260 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:51,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,262 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:51,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:51,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:51,266 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,266 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:51,267 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 20:55:51,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:51,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:51,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:51,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:51,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:51,267 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:51,285 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/73551f1efabc4bc297a0d92cc3a6ca13 2023-04-20 20:55:51,309 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/1ee186de1ec745fab6de4e412689ea34 2023-04-20 20:55:51,325 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/dca83cd15d2a45cface16d0248946f0d 2023-04-20 20:55:51,329 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/73551f1efabc4bc297a0d92cc3a6ca13 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/73551f1efabc4bc297a0d92cc3a6ca13 2023-04-20 20:55:51,332 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/73551f1efabc4bc297a0d92cc3a6ca13, entries=150, sequenceid=79, filesize=11.6 K 2023-04-20 20:55:51,333 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/1ee186de1ec745fab6de4e412689ea34 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/1ee186de1ec745fab6de4e412689ea34 2023-04-20 20:55:51,338 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/1ee186de1ec745fab6de4e412689ea34, entries=150, sequenceid=79, filesize=11.6 K 2023-04-20 20:55:51,339 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/dca83cd15d2a45cface16d0248946f0d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dca83cd15d2a45cface16d0248946f0d 2023-04-20 20:55:51,345 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dca83cd15d2a45cface16d0248946f0d, entries=150, sequenceid=79, filesize=11.6 K 2023-04-20 20:55:51,346 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 80ms, sequenceid=79, compaction requested=true 2023-04-20 20:55:51,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:51,346 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:51,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:51,346 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:51,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,351 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:51,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58050559[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:51,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:51,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,355 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:51,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:51,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:51,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:51,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,360 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:51,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:51,360 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:51,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:51,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:51,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:51,361 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:51,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,363 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,368 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:51,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:51,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,373 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:51,373 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:51,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:51,473 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:51,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:51,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:51,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:51,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:51,476 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:51,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:51,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:51,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:51,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:51,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:51,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:51,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:51,483 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,483 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:51,483 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:51,483 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:51,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,485 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,485 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:51,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47cb5afd[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:51,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:51,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,489 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:51,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,489 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:51,489 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:51,489 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:51,490 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:51,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:51,494 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:51,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:51,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:51,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:51,494 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:51,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,499 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,503 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,503 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,503 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,503 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:51,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:51,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,504 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:51,504 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:51,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:51,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 20:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:51,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:51,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/7833df0ef9bc4c5b887d911c9c047f30 2023-04-20 20:55:51,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/576f6a780dad4ead84ff81d518fd4922 2023-04-20 20:55:51,604 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:51,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:51,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:51,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:51,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:51,609 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:51,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:51,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:51,610 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,610 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,611 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:51,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:51,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:51,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:51,620 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,620 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:51,620 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:51,620 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:51,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/dadfc2bdd0564a33be2bdac1b7b75ec0 2023-04-20 20:55:51,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288c4816[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:51,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:51,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,627 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/7833df0ef9bc4c5b887d911c9c047f30 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/7833df0ef9bc4c5b887d911c9c047f30 2023-04-20 20:55:51,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:51,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,629 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/7833df0ef9bc4c5b887d911c9c047f30, entries=250, sequenceid=91, filesize=16.2 K 2023-04-20 20:55:51,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024151619, max timestamp=1682024151633Num uniques 859; 2023-04-20 20:55:51,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:51,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024151619, max timestamp=1682024151633Num uniques 859; 2023-04-20 20:55:51,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024151619, max timestamp=1682024151633Num uniques 859; 2023-04-20 20:55:51,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:51,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:51,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/576f6a780dad4ead84ff81d518fd4922 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/576f6a780dad4ead84ff81d518fd4922 2023-04-20 20:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:51,636 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:51,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:51,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:51,636 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:51,636 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:51,638 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,638 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,638 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/576f6a780dad4ead84ff81d518fd4922, entries=100, sequenceid=91, filesize=9.3 K 2023-04-20 20:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/dadfc2bdd0564a33be2bdac1b7b75ec0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dadfc2bdd0564a33be2bdac1b7b75ec0 2023-04-20 20:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dadfc2bdd0564a33be2bdac1b7b75ec0, entries=100, sequenceid=91, filesize=9.3 K 2023-04-20 20:55:51,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=46.96 KB/48090 for 8552cf5289bf62dd18e69f1f78b0b5fd in 66ms, sequenceid=91, compaction requested=true 2023-04-20 20:55:51,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:51,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:51,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:51,646 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:55:51,646 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:55:51,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:51,647 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,646 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,646 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:51,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,647 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:51,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:51,647 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,648 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:55:51,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:51,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-20 20:55:51,649 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:55:51,649 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:51,649 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fb91de30ec6140f6874774a3c0911c78, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a7b07ffb4a034b4f9a31d1fa11f3413f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86df1595b92948b3ab966e47f2abfa4b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/1ee186de1ec745fab6de4e412689ea34, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/576f6a780dad4ead84ff81d518fd4922] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=55.7 K 2023-04-20 20:55:51,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:51,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:51,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:51,649 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 73470 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:55:51,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:51,649 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:55:51,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:51,649 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,650 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4d25dcef4cc34626956fad317658ddc0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/96a5c966f6924e34ba673019e1436082, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a5ebe368ddf94ab99470bcd93bfa5f9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/73551f1efabc4bc297a0d92cc3a6ca13, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/7833df0ef9bc4c5b887d911c9c047f30] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=71.7 K 2023-04-20 20:55:51,650 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting fb91de30ec6140f6874774a3c0911c78, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682024148604 2023-04-20 20:55:51,650 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a7b07ffb4a034b4f9a31d1fa11f3413f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024148680 2023-04-20 20:55:51,651 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 86df1595b92948b3ab966e47f2abfa4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682024149790 2023-04-20 20:55:51,651 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 4d25dcef4cc34626956fad317658ddc0, keycount=350, bloomtype=ROW, size=20.8 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682024148603 2023-04-20 20:55:51,651 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 96a5c966f6924e34ba673019e1436082, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024148680 2023-04-20 20:55:51,651 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1ee186de1ec745fab6de4e412689ea34, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682024150440 2023-04-20 20:55:51,652 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a5ebe368ddf94ab99470bcd93bfa5f9d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682024149790 2023-04-20 20:55:51,652 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 576f6a780dad4ead84ff81d518fd4922, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682024151578 2023-04-20 20:55:51,652 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 73551f1efabc4bc297a0d92cc3a6ca13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682024150440 2023-04-20 20:55:51,653 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 7833df0ef9bc4c5b887d911c9c047f30, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682024151570 2023-04-20 20:55:51,678 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#402 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:51,682 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#403 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:51,693 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/9ea0f8dc98bd47bb8708f7d7d02b77d8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/9ea0f8dc98bd47bb8708f7d7d02b77d8 2023-04-20 20:55:51,699 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 9ea0f8dc98bd47bb8708f7d7d02b77d8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:51,699 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:51,699 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=11, startTime=1682024151646; duration=0sec 2023-04-20 20:55:51,699 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:51,699 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:55:51,701 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:55:51,701 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:55:51,701 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,701 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/ef8d8897201145b099236b9265d9a28d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70cc2a822968454780e3c3c53f18eef3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a0ae507575974742918043a4ca5a68ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dca83cd15d2a45cface16d0248946f0d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dadfc2bdd0564a33be2bdac1b7b75ec0] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=55.7 K 2023-04-20 20:55:51,701 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ef8d8897201145b099236b9265d9a28d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682024148604 2023-04-20 20:55:51,702 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 70cc2a822968454780e3c3c53f18eef3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024148680 2023-04-20 20:55:51,702 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a0ae507575974742918043a4ca5a68ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682024149790 2023-04-20 20:55:51,702 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting dca83cd15d2a45cface16d0248946f0d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682024150440 2023-04-20 20:55:51,703 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting dadfc2bdd0564a33be2bdac1b7b75ec0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682024151578 2023-04-20 20:55:51,712 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#404 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:51,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024151695, max timestamp=1682024151720Num uniques 859; 2023-04-20 20:55:51,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024151695, max timestamp=1682024151720Num uniques 859; 2023-04-20 20:55:51,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:51,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024151695, max timestamp=1682024151720Num uniques 859; 2023-04-20 20:55:51,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:51,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:51,748 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:51,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:51,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:51,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:51,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:51,758 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:51,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:51,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:51,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,761 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,762 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,762 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:51,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:51,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:51,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:51,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:51,766 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:51,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:51,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6ff5d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:51,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:51,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:51,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,771 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:51,772 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:51,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:51,779 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:51,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:51,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:51,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:51,779 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:51,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024211765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024211777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,791 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:51,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:51,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,795 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:51,795 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:51,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:51,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024211782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024211782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,865 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 20:55:51,895 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:51,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:51,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024211883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024211883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:51,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:51,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:51,900 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:51,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:51,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:51,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,903 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:51,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:51,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:51,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:51,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:51,906 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,906 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,907 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,907 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:51,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a90dd7[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:51,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:51,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:51,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:51,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:51,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024211900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:51,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024211900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:51,917 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:51,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:51,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:51,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:51,917 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:51,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:51,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:51,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:51,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:51,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:51,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:51,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:51,925 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:51,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:51,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:51,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:51,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:51,925 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:51,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,025 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:52,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:52,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:52,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:52,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:52,037 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:52,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:52,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:52,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,043 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,043 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:52,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:52,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:52,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:52,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:52,046 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:52,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da8bd8d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:52,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:52,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,059 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:52,060 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,060 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:52,068 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:52,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/121669bb5bd44d1b8db17b1584c229c0 2023-04-20 20:55:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:52,074 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:52,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:52,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:52,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:52,074 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:52,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:52,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:52,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:52,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/33c14dd6ea8246adabd79bbdafa7e764 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,088 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:52,088 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:52,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,102 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/fca170cb0b4a447b948571c236dcaf3a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fca170cb0b4a447b948571c236dcaf3a 2023-04-20 20:55:52,107 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into fca170cb0b4a447b948571c236dcaf3a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:52,107 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:52,107 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=11, startTime=1682024151646; duration=0sec 2023-04-20 20:55:52,107 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:52,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/6f332fb920e74b779a0caa96921c52c3 2023-04-20 20:55:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024212100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024212100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024212115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024212116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/121669bb5bd44d1b8db17b1584c229c0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/121669bb5bd44d1b8db17b1584c229c0 2023-04-20 20:55:52,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/121669bb5bd44d1b8db17b1584c229c0, entries=200, sequenceid=116, filesize=13.9 K 2023-04-20 20:55:52,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/33c14dd6ea8246adabd79bbdafa7e764 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/33c14dd6ea8246adabd79bbdafa7e764 2023-04-20 20:55:52,134 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/82b33407f2fb4c378e7871acf82bea5b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/82b33407f2fb4c378e7871acf82bea5b 2023-04-20 20:55:52,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/33c14dd6ea8246adabd79bbdafa7e764, entries=150, sequenceid=116, filesize=11.6 K 2023-04-20 20:55:52,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/6f332fb920e74b779a0caa96921c52c3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f332fb920e74b779a0caa96921c52c3 2023-04-20 20:55:52,138 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into 82b33407f2fb4c378e7871acf82bea5b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:52,138 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:52,138 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=11, startTime=1682024151646; duration=0sec 2023-04-20 20:55:52,138 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:52,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f332fb920e74b779a0caa96921c52c3, entries=150, sequenceid=116, filesize=11.6 K 2023-04-20 20:55:52,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for 8552cf5289bf62dd18e69f1f78b0b5fd in 495ms, sequenceid=116, compaction requested=false 2023-04-20 20:55:52,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:52,188 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:52,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:52,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:52,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:52,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:52,192 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,196 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,196 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:52,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:52,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:52,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:52,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:52,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:52,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:52,198 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,198 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:52,198 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-20 20:55:52,198 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:52,199 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:52,199 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:52,199 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:52,199 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:52,199 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:52,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:52,419 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:52,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024212431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024212436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024212438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024212438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024212538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024212541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024212548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024212548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,619 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/296fb75a6a6b4ad4bd00717f187f1fa4 2023-04-20 20:55:52,636 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/bc44adb7116b4c559047004684e08285 2023-04-20 20:55:52,647 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/f6e0ab3d5812465aaac2f9414300bfc2 2023-04-20 20:55:52,653 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/296fb75a6a6b4ad4bd00717f187f1fa4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/296fb75a6a6b4ad4bd00717f187f1fa4 2023-04-20 20:55:52,658 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/296fb75a6a6b4ad4bd00717f187f1fa4, entries=150, sequenceid=155, filesize=11.8 K 2023-04-20 20:55:52,659 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/bc44adb7116b4c559047004684e08285 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/bc44adb7116b4c559047004684e08285 2023-04-20 20:55:52,663 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/bc44adb7116b4c559047004684e08285, entries=150, sequenceid=155, filesize=11.8 K 2023-04-20 20:55:52,664 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/f6e0ab3d5812465aaac2f9414300bfc2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f6e0ab3d5812465aaac2f9414300bfc2 2023-04-20 20:55:52,669 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f6e0ab3d5812465aaac2f9414300bfc2, entries=150, sequenceid=155, filesize=11.8 K 2023-04-20 20:55:52,670 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 8552cf5289bf62dd18e69f1f78b0b5fd in 472ms, sequenceid=155, compaction requested=true 2023-04-20 20:55:52,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:52,670 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:52,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:52,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:52,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272e94d6[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:52,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:52,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,677 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:52,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:52,679 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:52,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:52,684 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:52,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:52,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:52,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:52,684 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:52,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,694 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,694 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,694 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,694 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,694 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:52,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,694 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:52,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:52,695 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:52,695 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:52,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:52,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:55:52,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:52,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:52,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:52,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:52,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:52,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:52,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/423f0779615a4e5999d419efe1326309 2023-04-20 20:55:52,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/4e458f4dbad8440fb065bcf269cb88d1 2023-04-20 20:55:52,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024212784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,795 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:52,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024212785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024212788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024212789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:52,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:52,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:52,804 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:52,804 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/bacecb747a324d1a8231c0af521afa02 2023-04-20 20:55:52,804 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,804 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:52,804 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:52,805 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,805 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,807 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:52,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:52,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:52,809 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,809 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:52,809 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,809 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:52,809 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:52,809 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:52,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/423f0779615a4e5999d419efe1326309 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/423f0779615a4e5999d419efe1326309 2023-04-20 20:55:52,811 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:52,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10159b79[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:52,814 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:52,814 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,815 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:52,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:52,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:52,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:52,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/423f0779615a4e5999d419efe1326309, entries=200, sequenceid=167, filesize=14.1 K 2023-04-20 20:55:52,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/4e458f4dbad8440fb065bcf269cb88d1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4e458f4dbad8440fb065bcf269cb88d1 2023-04-20 20:55:52,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4e458f4dbad8440fb065bcf269cb88d1, entries=150, sequenceid=167, filesize=11.8 K 2023-04-20 20:55:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/bacecb747a324d1a8231c0af521afa02 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/bacecb747a324d1a8231c0af521afa02 2023-04-20 20:55:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:52,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:52,822 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:52,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:52,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:52,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:52,822 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:52,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,825 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,826 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,828 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,828 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,829 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:52,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:52,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,829 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:52,829 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:52,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,833 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:52,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/bacecb747a324d1a8231c0af521afa02, entries=150, sequenceid=167, filesize=11.8 K 2023-04-20 20:55:52,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8552cf5289bf62dd18e69f1f78b0b5fd in 88ms, sequenceid=167, compaction requested=true 2023-04-20 20:55:52,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:52,837 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:52,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:52,840 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52746 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:52,840 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:55:52,840 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,840 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/9ea0f8dc98bd47bb8708f7d7d02b77d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/121669bb5bd44d1b8db17b1584c229c0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/296fb75a6a6b4ad4bd00717f187f1fa4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/423f0779615a4e5999d419efe1326309] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=51.5 K 2023-04-20 20:55:52,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:52,841 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:52,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:52,842 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9ea0f8dc98bd47bb8708f7d7d02b77d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682024150447 2023-04-20 20:55:52,842 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 121669bb5bd44d1b8db17b1584c229c0, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682024151622 2023-04-20 20:55:52,842 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 296fb75a6a6b4ad4bd00717f187f1fa4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1682024151763 2023-04-20 20:55:52,843 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:52,843 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:55:52,843 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,843 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fca170cb0b4a447b948571c236dcaf3a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/33c14dd6ea8246adabd79bbdafa7e764, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/bc44adb7116b4c559047004684e08285, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4e458f4dbad8440fb065bcf269cb88d1] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=46.9 K 2023-04-20 20:55:52,843 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting fca170cb0b4a447b948571c236dcaf3a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682024150447 2023-04-20 20:55:52,843 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 423f0779615a4e5999d419efe1326309, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682024152419 2023-04-20 20:55:52,843 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 33c14dd6ea8246adabd79bbdafa7e764, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682024151637 2023-04-20 20:55:52,844 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting bc44adb7116b4c559047004684e08285, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1682024151763 2023-04-20 20:55:52,844 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4e458f4dbad8440fb065bcf269cb88d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682024152419 2023-04-20 20:55:52,853 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:52,855 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#414 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:52,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024152781, max timestamp=1682024152859Num uniques 859; 2023-04-20 20:55:52,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:52,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024152781, max timestamp=1682024152859Num uniques 859; 2023-04-20 20:55:52,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:52,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024152781, max timestamp=1682024152859Num uniques 859; 2023-04-20 20:55:52,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:55:52,873 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/6b993e7804cc4768b6acaabe968770ab as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6b993e7804cc4768b6acaabe968770ab 2023-04-20 20:55:52,879 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/8ed404851810491da56fb2e5292c0b35 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8ed404851810491da56fb2e5292c0b35 2023-04-20 20:55:52,883 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 8ed404851810491da56fb2e5292c0b35(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:52,883 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:52,883 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=12, startTime=1682024152837; duration=0sec 2023-04-20 20:55:52,883 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:52,883 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:52,884 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:52,885 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:55:52,885 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,885 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/82b33407f2fb4c378e7871acf82bea5b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f332fb920e74b779a0caa96921c52c3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f6e0ab3d5812465aaac2f9414300bfc2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/bacecb747a324d1a8231c0af521afa02] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=46.9 K 2023-04-20 20:55:52,885 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 82b33407f2fb4c378e7871acf82bea5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682024150447 2023-04-20 20:55:52,885 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6f332fb920e74b779a0caa96921c52c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682024151637 2023-04-20 20:55:52,886 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f6e0ab3d5812465aaac2f9414300bfc2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1682024151763 2023-04-20 20:55:52,886 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting bacecb747a324d1a8231c0af521afa02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682024152419 2023-04-20 20:55:52,894 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#415 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:52,896 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 6b993e7804cc4768b6acaabe968770ab(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:52,896 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:52,896 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=12, startTime=1682024152841; duration=0sec 2023-04-20 20:55:52,896 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:52,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-20 20:55:52,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:52,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:52,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:52,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:52,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:52,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:52,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:52,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/206162eac6014ac7bd240709a9f065af 2023-04-20 20:55:52,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682024152914, max timestamp=1682024152927Num uniques 859; 2023-04-20 20:55:52,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:52,929 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:52,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:52,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024152914, max timestamp=1682024152927Num uniques 859; 2023-04-20 20:55:52,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:52,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024152914, max timestamp=1682024152927Num uniques 859; 2023-04-20 20:55:52,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:55:52,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:52,934 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:52,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:52,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:52,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,936 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:52,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:52,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:52,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:52,944 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:52,944 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:52,944 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:52,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:52,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:52,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:52,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:52,956 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,962 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:52,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286afb19[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:52,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:52,963 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/8999ec63d2d74cafb57be2b4a74e5e65 2023-04-20 20:55:52,965 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:52,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:52,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024212957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024212960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:52,971 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:52,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:52,972 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:52,972 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:52,972 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:52,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:52,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:52,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024212962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024212963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:52,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024212964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:52,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:52,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:52,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:52,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:52,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:52,979 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:52,979 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:53,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024213070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024213070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,080 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:53,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:53,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:53,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:53,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:53,083 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:53,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024213078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,085 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:53,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:53,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:53,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:53,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:53,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024213078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:53,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024213079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:53,095 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@866f4c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:53,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:53,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,099 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:53,099 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:53,099 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:53,099 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,099 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:53,101 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,101 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:53,105 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:53,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:53,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:53,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:53,105 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:53,106 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,106 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:53,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,106 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,107 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,111 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,122 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:53,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:53,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:53,122 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:53,122 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:53,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:53,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:53,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:53,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,124 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,124 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,124 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:53,223 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:53,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:53,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:53,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:53,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:53,226 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:53,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:53,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:53,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,229 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:53,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:53,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,229 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:53,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:53,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:53,230 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:53,230 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:53,231 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:53,231 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:53,232 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ddd3ae[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:53,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:53,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,237 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,237 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:53,237 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:53,237 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:53,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:53,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:53,241 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:53,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:53,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:53,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:53,241 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:53,245 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,245 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,245 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:53,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,247 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:53,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:53,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,251 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,251 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:53,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:53,251 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:53,251 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:53,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:53,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:53,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024213278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024213278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024213286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024213290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024213291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,330 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/e99073d26bf34a119a3bd25d1e729c5e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e99073d26bf34a119a3bd25d1e729c5e 2023-04-20 20:55:53,334 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into e99073d26bf34a119a3bd25d1e729c5e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:53,334 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:53,334 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=12, startTime=1682024152841; duration=0sec 2023-04-20 20:55:53,335 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:53,352 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:53,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:53,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:53,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:53,360 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:53,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:53,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:53,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,362 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:53,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:53,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:53,366 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:53,367 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:53,367 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:53,367 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:53,367 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:53,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,369 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:53,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2f44ff[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:53,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:53,372 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,374 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:53,374 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,374 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:53,375 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:53,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:53,379 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:53,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:53,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:53,379 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:53,379 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:53,381 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:53,382 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,382 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:53,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,382 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:53,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:53,384 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:53,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d09515dacf904cd890b0821d0b172b4b 2023-04-20 20:55:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:53,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,393 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:53,394 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,394 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:53,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:53,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:53,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:53,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:53,394 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:53,394 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:53,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/206162eac6014ac7bd240709a9f065af as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/206162eac6014ac7bd240709a9f065af 2023-04-20 20:55:53,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/206162eac6014ac7bd240709a9f065af, entries=200, sequenceid=194, filesize=14.1 K 2023-04-20 20:55:53,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/8999ec63d2d74cafb57be2b4a74e5e65 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/8999ec63d2d74cafb57be2b4a74e5e65 2023-04-20 20:55:53,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/8999ec63d2d74cafb57be2b4a74e5e65, entries=150, sequenceid=194, filesize=11.8 K 2023-04-20 20:55:53,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d09515dacf904cd890b0821d0b172b4b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d09515dacf904cd890b0821d0b172b4b 2023-04-20 20:55:53,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d09515dacf904cd890b0821d0b172b4b, entries=150, sequenceid=194, filesize=11.8 K 2023-04-20 20:55:53,418 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for 8552cf5289bf62dd18e69f1f78b0b5fd in 522ms, sequenceid=194, compaction requested=false 2023-04-20 20:55:53,418 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:53,494 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:53,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:53,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:53,499 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:53,499 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,499 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:53,499 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:53,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,500 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,504 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:53,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:53,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:53,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:53,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:53,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:53,506 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:53,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:53,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:53,512 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-20 20:55:53,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:53,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:53,512 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:53,533 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/070c95846a524ec4a0d2a81f6b55493d 2023-04-20 20:55:53,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:53,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:53,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024213598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024213600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024213603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024213605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024213610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024213710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024213714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024213714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024213917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024213925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:53,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024213925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:53,966 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/712886058ef846e4b87c4d9577d9553d 2023-04-20 20:55:53,982 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/6f47ce5f16094bb3a4cde85abf6625e3 2023-04-20 20:55:53,987 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/070c95846a524ec4a0d2a81f6b55493d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/070c95846a524ec4a0d2a81f6b55493d 2023-04-20 20:55:53,992 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/070c95846a524ec4a0d2a81f6b55493d, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 20:55:53,993 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/712886058ef846e4b87c4d9577d9553d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/712886058ef846e4b87c4d9577d9553d 2023-04-20 20:55:53,997 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/712886058ef846e4b87c4d9577d9553d, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 20:55:53,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/6f47ce5f16094bb3a4cde85abf6625e3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f47ce5f16094bb3a4cde85abf6625e3 2023-04-20 20:55:54,001 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f47ce5f16094bb3a4cde85abf6625e3, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 20:55:54,004 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 8552cf5289bf62dd18e69f1f78b0b5fd in 492ms, sequenceid=230, compaction requested=true 2023-04-20 20:55:54,004 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:54,004 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:54,004 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:54,007 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:54,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10e2c830[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:54,011 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:54,011 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,012 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:54,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:54,013 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:54,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:54,018 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:54,020 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:54,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:54,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:54,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:54,021 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:54,023 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,023 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,023 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,027 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,027 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:54,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,029 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:54,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:54,029 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:54,029 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:54,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:54,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 20:55:54,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:54,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:54,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:54,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:54,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:54,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:54,129 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:54,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:54,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:54,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:54,138 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:54,138 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,139 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:54,139 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:54,139 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,139 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,140 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,140 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:54,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:54,149 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ad396f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:54,151 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:54,151 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,152 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:54,153 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,153 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:54,154 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:54,156 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:54,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:54,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:54,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:54,157 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:54,158 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,161 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,164 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,164 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,164 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,164 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,164 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:54,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:54,164 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,164 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,165 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,165 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,165 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,165 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:54,165 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:54,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024154198, max timestamp=1682024154218Num uniques 859; 2023-04-20 20:55:54,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024154198, max timestamp=1682024154218Num uniques 859; 2023-04-20 20:55:54,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:54,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024154198, max timestamp=1682024154218Num uniques 859; 2023-04-20 20:55:54,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:54,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:55:54,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024214242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024214244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024214250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024214251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,265 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024214252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:54,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:54,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:54,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:54,269 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:54,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:54,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:54,270 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,270 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:54,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:54,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:54,273 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,274 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:54,274 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:54,274 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:54,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:54,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:54,274 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,275 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e50ce17[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:54,277 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:54,277 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:54,280 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,280 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:54,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:54,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,284 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:54,287 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:54,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:54,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:54,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:54,287 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:54,288 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,288 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,288 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,292 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:54,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:54,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:54,296 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:54,296 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:54,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024214353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024214354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024214358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024214366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024214367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,396 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:54,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:54,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:54,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:54,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:54,399 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:54,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:54,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:54,400 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,400 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:54,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:54,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:54,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:54,403 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:54,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,405 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4d799e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:54,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:54,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:54,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,410 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:54,411 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:54,413 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:54,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:54,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:54,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:54,413 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:54,418 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,418 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,418 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,423 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:54,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:54,424 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:54,424 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:54,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/5c610fbf8b5d496b98a9280213171505 2023-04-20 20:55:54,524 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:54,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:54,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:54,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:54,528 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:54,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:54,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:54,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,533 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,533 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,533 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,533 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:54,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:54,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:54,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:54,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:54,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/a32aad2dbe9f462ea43a603b4278f822 2023-04-20 20:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@de8938f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:54,553 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:54,553 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:54,557 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:54,557 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,557 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,557 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:54,561 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:54,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:54,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:54,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:54,562 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:54,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,565 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:54,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:54,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,570 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:54,570 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:54,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/576da2d64ba349989478876ead3b433d 2023-04-20 20:55:54,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024214561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024214561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/5c610fbf8b5d496b98a9280213171505 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5c610fbf8b5d496b98a9280213171505 2023-04-20 20:55:54,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024214569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024214571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024214571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5c610fbf8b5d496b98a9280213171505, entries=150, sequenceid=241, filesize=11.8 K 2023-04-20 20:55:54,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/a32aad2dbe9f462ea43a603b4278f822 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a32aad2dbe9f462ea43a603b4278f822 2023-04-20 20:55:54,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a32aad2dbe9f462ea43a603b4278f822, entries=100, sequenceid=241, filesize=9.4 K 2023-04-20 20:55:54,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/576da2d64ba349989478876ead3b433d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/576da2d64ba349989478876ead3b433d 2023-04-20 20:55:54,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/576da2d64ba349989478876ead3b433d, entries=100, sequenceid=241, filesize=9.4 K 2023-04-20 20:55:54,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8552cf5289bf62dd18e69f1f78b0b5fd in 475ms, sequenceid=241, compaction requested=true 2023-04-20 20:55:54,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:54,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:54,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:54,589 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:54,588 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:54,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:54,590 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46058 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:54,590 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:55:54,590 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50838 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:54,590 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,590 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:55:54,590 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6b993e7804cc4768b6acaabe968770ab, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/8999ec63d2d74cafb57be2b4a74e5e65, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/712886058ef846e4b87c4d9577d9553d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a32aad2dbe9f462ea43a603b4278f822] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=45.0 K 2023-04-20 20:55:54,590 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,590 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8ed404851810491da56fb2e5292c0b35, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/206162eac6014ac7bd240709a9f065af, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/070c95846a524ec4a0d2a81f6b55493d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5c610fbf8b5d496b98a9280213171505] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=49.6 K 2023-04-20 20:55:54,590 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 6b993e7804cc4768b6acaabe968770ab, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682024152419 2023-04-20 20:55:54,592 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8ed404851810491da56fb2e5292c0b35, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682024152419 2023-04-20 20:55:54,592 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8999ec63d2d74cafb57be2b4a74e5e65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682024152787 2023-04-20 20:55:54,592 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 206162eac6014ac7bd240709a9f065af, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682024152787 2023-04-20 20:55:54,592 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 712886058ef846e4b87c4d9577d9553d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682024152959 2023-04-20 20:55:54,592 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 070c95846a524ec4a0d2a81f6b55493d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682024152959 2023-04-20 20:55:54,592 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a32aad2dbe9f462ea43a603b4278f822, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682024153595 2023-04-20 20:55:54,593 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5c610fbf8b5d496b98a9280213171505, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682024153595 2023-04-20 20:55:54,601 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:54,603 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#426 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:54,615 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/659780e686bc4742a587ef598b342275 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/659780e686bc4742a587ef598b342275 2023-04-20 20:55:54,616 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/2b75c60b6a8a45088d76a960ac0dc3c0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2b75c60b6a8a45088d76a960ac0dc3c0 2023-04-20 20:55:54,620 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 2b75c60b6a8a45088d76a960ac0dc3c0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:54,620 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 659780e686bc4742a587ef598b342275(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:54,620 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:54,620 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:54,620 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=12, startTime=1682024154588; duration=0sec 2023-04-20 20:55:54,620 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=12, startTime=1682024154588; duration=0sec 2023-04-20 20:55:54,621 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:54,621 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:54,621 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:54,621 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46058 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:54,622 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:55:54,622 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,622 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e99073d26bf34a119a3bd25d1e729c5e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d09515dacf904cd890b0821d0b172b4b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f47ce5f16094bb3a4cde85abf6625e3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/576da2d64ba349989478876ead3b433d] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=45.0 K 2023-04-20 20:55:54,622 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e99073d26bf34a119a3bd25d1e729c5e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682024152419 2023-04-20 20:55:54,622 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d09515dacf904cd890b0821d0b172b4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682024152787 2023-04-20 20:55:54,622 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6f47ce5f16094bb3a4cde85abf6625e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682024152959 2023-04-20 20:55:54,623 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 576da2d64ba349989478876ead3b433d, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682024153595 2023-04-20 20:55:54,628 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#427 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:54,637 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/f362317780d6489e93474bb3f394a9b9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f362317780d6489e93474bb3f394a9b9 2023-04-20 20:55:54,641 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into f362317780d6489e93474bb3f394a9b9(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:54,641 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:54,641 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=12, startTime=1682024154589; duration=0sec 2023-04-20 20:55:54,641 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:54,670 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:54,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:54,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:54,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:54,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:54,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:54,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:54,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:54,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:54,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:54,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:54,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:54,678 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 20:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:54,678 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:54,679 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:54,679 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:54,693 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/3444b46f57e74eeabb23bc1be339d2d8 2023-04-20 20:55:54,702 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/38f6cd5269454a5da79e63701f3fe9fe 2023-04-20 20:55:54,713 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/a8c6241040f446738765f5179ad909c8 2023-04-20 20:55:54,718 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/3444b46f57e74eeabb23bc1be339d2d8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/3444b46f57e74eeabb23bc1be339d2d8 2023-04-20 20:55:54,721 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/3444b46f57e74eeabb23bc1be339d2d8, entries=150, sequenceid=279, filesize=11.9 K 2023-04-20 20:55:54,723 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/38f6cd5269454a5da79e63701f3fe9fe as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/38f6cd5269454a5da79e63701f3fe9fe 2023-04-20 20:55:54,735 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/38f6cd5269454a5da79e63701f3fe9fe, entries=150, sequenceid=279, filesize=11.9 K 2023-04-20 20:55:54,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/a8c6241040f446738765f5179ad909c8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a8c6241040f446738765f5179ad909c8 2023-04-20 20:55:54,741 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a8c6241040f446738765f5179ad909c8, entries=150, sequenceid=279, filesize=11.9 K 2023-04-20 20:55:54,742 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 64ms, sequenceid=279, compaction requested=false 2023-04-20 20:55:54,742 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:54,742 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:54,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,744 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:54,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4744b51c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:54,747 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:54,747 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,749 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:54,750 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:54,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:54,760 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:54,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:54,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:54,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:54,760 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:54,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,763 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,767 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:54,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:54,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,767 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,767 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:54,767 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:54,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,867 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:54,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:54,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:54,871 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:54,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:54,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:54,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,873 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:54,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:54,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:54,876 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,876 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:54,877 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-20 20:55:54,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:54,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:54,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:54,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:54,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:54,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:54,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:54,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:54,893 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/5d98ecda38e84b869354d3df2fec3817 2023-04-20 20:55:54,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024154891, max timestamp=1682024154904Num uniques 859; 2023-04-20 20:55:54,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:55:54,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024154891, max timestamp=1682024154904Num uniques 859; 2023-04-20 20:55:54,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682024154891, max timestamp=1682024154904Num uniques 859; 2023-04-20 20:55:54,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:55:54,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:55:54,915 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/73d3604b42de4a6f989ca65e7844f81c 2023-04-20 20:55:54,925 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/3ce6a26eda374ad59a6bedc0315e6e52 2023-04-20 20:55:54,929 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/5d98ecda38e84b869354d3df2fec3817 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5d98ecda38e84b869354d3df2fec3817 2023-04-20 20:55:54,934 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5d98ecda38e84b869354d3df2fec3817, entries=100, sequenceid=284, filesize=9.5 K 2023-04-20 20:55:54,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682024154920, max timestamp=1682024154934Num uniques 1168; 2023-04-20 20:55:54,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682024154920, max timestamp=1682024154934Num uniques 1168; 2023-04-20 20:55:54,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-20 20:55:54,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-20 20:55:54,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682024154920, max timestamp=1682024154934Num uniques 1168; 2023-04-20 20:55:54,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-20 20:55:54,940 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/73d3604b42de4a6f989ca65e7844f81c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/73d3604b42de4a6f989ca65e7844f81c 2023-04-20 20:55:54,944 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/73d3604b42de4a6f989ca65e7844f81c, entries=100, sequenceid=284, filesize=9.5 K 2023-04-20 20:55:54,945 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/3ce6a26eda374ad59a6bedc0315e6e52 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3ce6a26eda374ad59a6bedc0315e6e52 2023-04-20 20:55:54,951 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3ce6a26eda374ad59a6bedc0315e6e52, entries=100, sequenceid=284, filesize=9.5 K 2023-04-20 20:55:54,952 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=87.22 KB/89310 for 8552cf5289bf62dd18e69f1f78b0b5fd in 75ms, sequenceid=284, compaction requested=true 2023-04-20 20:55:54,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:54,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:54,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:54,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=93.93 KB heapSize=255.23 KB 2023-04-20 20:55:54,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:54,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:54,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:54,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:54,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:54,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:55:54,959 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:54,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ffcfb95[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:54,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:54,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,963 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:54,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:54,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:54,965 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:54,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:54,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:54,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:54,965 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:54,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:54,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:54,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,969 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:54,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:54,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:54,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:54,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:54,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:54,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:54,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:54,976 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:54,977 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:54,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:54,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:54,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:54,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024214977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024214985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024214985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024214987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:54,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024214987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,077 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:55,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:55,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:55,081 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:55,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:55,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:55,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,083 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:55,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:55,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:55,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:55,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:55,085 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:55,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f1df87d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:55,090 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:55,090 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,092 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:55,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:55,094 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:55,094 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,094 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024215088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024215089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:55,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:55,097 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:55,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:55,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:55,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:55,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,097 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:55,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024215093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024215093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024215093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,099 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,100 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,102 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,102 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,102 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:55,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,103 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:55,103 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:55,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,203 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:55,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:55,206 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:55,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:55,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:55,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:55,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:55,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:55,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:55,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:55,220 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,220 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:55,220 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:55,220 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,222 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762060fa[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:55,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:55,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:55,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,227 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:55,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:55,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,229 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:55,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:55,230 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:55,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:55,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:55,230 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:55,230 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:55,232 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,233 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,233 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,237 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:55,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:55,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,239 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:55,240 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:55,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,240 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024215297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024215297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024215298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024215299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024215300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,340 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:55,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:55,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:55,343 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:55,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:55,343 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:55,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,344 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,345 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,345 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:55,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:55,347 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,347 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:55,347 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:55,347 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,349 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:55,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142ed777[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:55,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:55,353 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,354 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:55,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:55,355 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:55,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:55,356 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:55,360 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:55,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:55,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:55,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:55,361 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:55,362 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,362 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,362 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,363 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,364 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,367 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,367 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:55,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:55,367 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,367 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:55,368 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:55,368 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:55,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,368 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/bfce66a1a2de46d29ccdf4ddbea70c43 2023-04-20 20:55:55,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/c522b7879e544951a9ae2d48e2e72416 2023-04-20 20:55:55,468 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:55,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:55,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:55,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:55,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:55,472 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:55,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:55,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:55,472 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:55,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:55,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:55,479 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:55,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cb6b634[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:55,484 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:55,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,486 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:55,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:55,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,486 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:55,486 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:55,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:55,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:55,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:55,492 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:55,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:55,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:55,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:55,492 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:55,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,493 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,494 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,495 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:55,497 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:55,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:55,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,498 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:55,498 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:55,598 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:55,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:55,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:55,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:55,602 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:55,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:55,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:55,602 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,605 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:55,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:55,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:55,606 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,606 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:55,607 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:55,607 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:55,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024215601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,609 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024215603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024215606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024215606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:55,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024215607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:55,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb87dcf[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:55,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:55,612 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:55,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:55,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:55,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:55,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:55,615 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,615 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:55,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:55,618 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:55,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:55,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:55,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:55,619 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:55,620 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,620 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,620 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,627 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:55,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:55,627 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:55,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:55,627 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:55,727 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:55,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:55,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:55,730 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:55,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:55,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:55,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:55,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:55,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:55,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:55,736 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:55,740 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,740 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9eaf512[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:55,742 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:55,742 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,746 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:55,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:55,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:55,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:55,750 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:55,753 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:55,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:55,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:55,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:55,754 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:55,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:55,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:55,757 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:55,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:55,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:55,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:55,762 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:55,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:55,763 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,763 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:55,763 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:55,763 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,763 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:55,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/2ea16a7317644339b83817f8cb026ee1 2023-04-20 20:55:55,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/bfce66a1a2de46d29ccdf4ddbea70c43 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/bfce66a1a2de46d29ccdf4ddbea70c43 2023-04-20 20:55:55,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/bfce66a1a2de46d29ccdf4ddbea70c43, entries=200, sequenceid=330, filesize=14.3 K 2023-04-20 20:55:55,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/c522b7879e544951a9ae2d48e2e72416 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c522b7879e544951a9ae2d48e2e72416 2023-04-20 20:55:55,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c522b7879e544951a9ae2d48e2e72416, entries=150, sequenceid=330, filesize=11.9 K 2023-04-20 20:55:55,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/2ea16a7317644339b83817f8cb026ee1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2ea16a7317644339b83817f8cb026ee1 2023-04-20 20:55:55,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2ea16a7317644339b83817f8cb026ee1, entries=150, sequenceid=330, filesize=11.9 K 2023-04-20 20:55:55,839 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~255.19 KB/261312, currentSize=93.93 KB/96180 for 8552cf5289bf62dd18e69f1f78b0b5fd in 881ms, sequenceid=330, compaction requested=true 2023-04-20 20:55:55,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:55,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:55,839 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:55,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:55,839 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:55,840 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:55,841 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49034 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:55,841 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:55:55,841 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,841 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2b75c60b6a8a45088d76a960ac0dc3c0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/3444b46f57e74eeabb23bc1be339d2d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5d98ecda38e84b869354d3df2fec3817, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/bfce66a1a2de46d29ccdf4ddbea70c43] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=47.9 K 2023-04-20 20:55:55,842 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:55,842 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:55:55,842 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,842 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/659780e686bc4742a587ef598b342275, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/38f6cd5269454a5da79e63701f3fe9fe, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/73d3604b42de4a6f989ca65e7844f81c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c522b7879e544951a9ae2d48e2e72416] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=45.5 K 2023-04-20 20:55:55,842 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2b75c60b6a8a45088d76a960ac0dc3c0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682024152959 2023-04-20 20:55:55,842 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 659780e686bc4742a587ef598b342275, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682024152959 2023-04-20 20:55:55,843 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 3444b46f57e74eeabb23bc1be339d2d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682024154249 2023-04-20 20:55:55,843 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 38f6cd5269454a5da79e63701f3fe9fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682024154249 2023-04-20 20:55:55,843 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5d98ecda38e84b869354d3df2fec3817, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682024154875 2023-04-20 20:55:55,844 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 73d3604b42de4a6f989ca65e7844f81c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682024154875 2023-04-20 20:55:55,844 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting bfce66a1a2de46d29ccdf4ddbea70c43, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682024154943 2023-04-20 20:55:55,844 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c522b7879e544951a9ae2d48e2e72416, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682024154943 2023-04-20 20:55:55,852 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#437 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:55,854 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:55,863 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:55,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:55,871 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/216b4dc275e445409a1c0efb5c369432 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/216b4dc275e445409a1c0efb5c369432 2023-04-20 20:55:55,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:55,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:55,872 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:55,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:55,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:55,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,875 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:55,875 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:55,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:55,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:55,877 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 216b4dc275e445409a1c0efb5c369432(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:55,877 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:55,877 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=12, startTime=1682024155839; duration=0sec 2023-04-20 20:55:55,877 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:55,877 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:55,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:55,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:55,878 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:55,878 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:55:55,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,879 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:55,879 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f362317780d6489e93474bb3f394a9b9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a8c6241040f446738765f5179ad909c8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3ce6a26eda374ad59a6bedc0315e6e52, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2ea16a7317644339b83817f8cb026ee1] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=45.5 K 2023-04-20 20:55:55,879 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:55,879 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f362317780d6489e93474bb3f394a9b9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1682024152959 2023-04-20 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:55,879 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:55,879 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a8c6241040f446738765f5179ad909c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1682024154249 2023-04-20 20:55:55,880 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3ce6a26eda374ad59a6bedc0315e6e52, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682024154875 2023-04-20 20:55:55,880 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2ea16a7317644339b83817f8cb026ee1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682024154943 2023-04-20 20:55:55,893 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:55,900 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/16c165d856c84451b60a18b686cf7328 2023-04-20 20:55:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:56,116 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:56,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024216128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024216131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39986 deadline: 1682024216132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024216134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40066 deadline: 1682024216134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1682024216235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024216235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024216239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024216240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024216240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,270 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/b976904139e348f989af70350deccec8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b976904139e348f989af70350deccec8 2023-04-20 20:55:56,274 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into b976904139e348f989af70350deccec8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:56,274 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:56,274 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=12, startTime=1682024155839; duration=0sec 2023-04-20 20:55:56,274 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:56,316 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/834cb95492574e61bcdae4797940b377 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/834cb95492574e61bcdae4797940b377 2023-04-20 20:55:56,320 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into 834cb95492574e61bcdae4797940b377(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:56,320 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:56,320 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=12, startTime=1682024155840; duration=0sec 2023-04-20 20:55:56,320 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:56,322 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/821dc1bc385048b69b9ec70ebc32b349 2023-04-20 20:55:56,333 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/853b4bf3959e47afbaceeacba0feac26 2023-04-20 20:55:56,337 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/16c165d856c84451b60a18b686cf7328 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/16c165d856c84451b60a18b686cf7328 2023-04-20 20:55:56,342 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/16c165d856c84451b60a18b686cf7328, entries=100, sequenceid=348, filesize=9.5 K 2023-04-20 20:55:56,343 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/821dc1bc385048b69b9ec70ebc32b349 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/821dc1bc385048b69b9ec70ebc32b349 2023-04-20 20:55:56,349 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/821dc1bc385048b69b9ec70ebc32b349, entries=100, sequenceid=348, filesize=9.5 K 2023-04-20 20:55:56,350 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/853b4bf3959e47afbaceeacba0feac26 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/853b4bf3959e47afbaceeacba0feac26 2023-04-20 20:55:56,352 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/853b4bf3959e47afbaceeacba0feac26, entries=100, sequenceid=348, filesize=9.5 K 2023-04-20 20:55:56,353 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 8552cf5289bf62dd18e69f1f78b0b5fd in 474ms, sequenceid=348, compaction requested=false 2023-04-20 20:55:56,353 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:56,353 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:56,353 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:56,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:56,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b9d1e37[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:56,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:56,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,362 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:56,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:56,362 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:56,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,364 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:56,366 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:56,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:56,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:56,366 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:56,366 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:56,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,369 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,369 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,370 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,372 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:56,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:56,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,373 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:56,373 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:56,373 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,373 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:56,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:55:56,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:56,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:56,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:56,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:56,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:56,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:56,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024216459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024216460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024216461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024216461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024216462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,473 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:56,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:56,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:56,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:56,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:56,476 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:56,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:56,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:56,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:56,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:56,481 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,481 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:56,481 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:56,481 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:56,481 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:56,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,483 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,483 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:56,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72030266[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:56,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:56,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,486 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:56,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:56,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:56,492 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:56,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:56,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:56,492 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:56,492 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,495 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:56,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,499 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,499 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:56,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:56,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,499 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,499 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:56,499 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:56,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024216567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024216570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40004 deadline: 1682024216570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024216571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024216573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,599 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:56,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:56,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:56,603 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:56,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:56,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:56,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,608 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,608 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,608 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,608 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:56,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:56,610 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,610 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:56,610 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:56,610 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:56,612 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d6a4fec[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:56,615 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:56,615 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,616 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:56,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:56,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:56,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:56,621 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:56,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:56,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:56,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:56,621 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:56,625 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,625 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,625 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,625 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:56,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:56,630 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,631 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:56,631 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:56,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,631 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,731 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:56,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:56,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:56,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:56,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:56,735 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:56,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:56,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:56,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,736 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,737 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,737 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:56,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:56,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:56,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:56,739 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:56,741 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13f67136[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:56,742 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:56,742 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,744 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:56,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:56,749 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:56,751 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:56,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:56,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:56,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:56,751 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:56,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:56,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:56,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:56,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,757 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:56,757 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:56,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024216770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024216778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024216778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024216780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024216779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,857 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:56,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/86c82e6052804af58c04adec4511cfc1 2023-04-20 20:55:56,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:56,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:56,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:56,861 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:56,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:56,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:56,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,864 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,864 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:56,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:56,866 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,866 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:56,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:56,867 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:56,869 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,869 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/6ff18b08a25a44039626aa59bab7e024 2023-04-20 20:55:56,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1019b9d9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:56,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:56,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,874 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:56,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:56,876 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:56,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:56,879 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:56,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:56,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:56,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:56,879 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:56,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d0ffc1c9357d47778b91725005460aba 2023-04-20 20:55:56,882 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:56,882 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,883 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,883 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/86c82e6052804af58c04adec4511cfc1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/86c82e6052804af58c04adec4511cfc1 2023-04-20 20:55:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:56,885 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:56,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:56,889 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:56,889 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:56,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:56,889 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/86c82e6052804af58c04adec4511cfc1, entries=250, sequenceid=368, filesize=16.7 K 2023-04-20 20:55:56,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/6ff18b08a25a44039626aa59bab7e024 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6ff18b08a25a44039626aa59bab7e024 2023-04-20 20:55:56,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6ff18b08a25a44039626aa59bab7e024, entries=150, sequenceid=368, filesize=11.9 K 2023-04-20 20:55:56,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d0ffc1c9357d47778b91725005460aba as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d0ffc1c9357d47778b91725005460aba 2023-04-20 20:55:56,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d0ffc1c9357d47778b91725005460aba, entries=150, sequenceid=368, filesize=11.9 K 2023-04-20 20:55:56,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 8552cf5289bf62dd18e69f1f78b0b5fd in 452ms, sequenceid=368, compaction requested=true 2023-04-20 20:55:56,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:56,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:56,900 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:56,900 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:56,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:56,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:56,901 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39574 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:56,901 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:55:56,901 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:56,901 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,901 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:55:56,901 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b976904139e348f989af70350deccec8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/16c165d856c84451b60a18b686cf7328, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/86c82e6052804af58c04adec4511cfc1] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=38.6 K 2023-04-20 20:55:56,901 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,901 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/216b4dc275e445409a1c0efb5c369432, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/821dc1bc385048b69b9ec70ebc32b349, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6ff18b08a25a44039626aa59bab7e024] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=33.9 K 2023-04-20 20:55:56,901 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b976904139e348f989af70350deccec8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682024154943 2023-04-20 20:55:56,901 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 216b4dc275e445409a1c0efb5c369432, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682024154943 2023-04-20 20:55:56,902 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 16c165d856c84451b60a18b686cf7328, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682024154983 2023-04-20 20:55:56,902 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 821dc1bc385048b69b9ec70ebc32b349, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682024154983 2023-04-20 20:55:56,902 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 86c82e6052804af58c04adec4511cfc1, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682024156131 2023-04-20 20:55:56,902 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 6ff18b08a25a44039626aa59bab7e024, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682024156133 2023-04-20 20:55:56,908 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:56,909 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#447 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:56,989 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:56,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:56,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:56,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:56,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:56,992 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:56,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:56,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:56,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,994 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:56,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:56,995 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:56,995 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:56,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:56,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:56,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:56,996 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 20:55:56,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:56,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:56,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:56,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:56,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:56,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:57,010 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/736e0fb3e9e14383a36ac00343f51c01 2023-04-20 20:55:57,031 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/d2a02cf52594401d807f5eed5988e91f 2023-04-20 20:55:57,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:57,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:57,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024217104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024217104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024217105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024217111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024217112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024217217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024217217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024217219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024217218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024217222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,326 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/39554c9e31394031bd312bb466a08cc3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/39554c9e31394031bd312bb466a08cc3 2023-04-20 20:55:57,330 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/2cf1c245a2ed4f8cac38068c65d87775 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2cf1c245a2ed4f8cac38068c65d87775 2023-04-20 20:55:57,331 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 39554c9e31394031bd312bb466a08cc3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:57,331 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:57,331 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=13, startTime=1682024156899; duration=0sec 2023-04-20 20:55:57,331 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:57,331 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:55:57,332 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:55:57,332 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:55:57,332 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:57,332 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/834cb95492574e61bcdae4797940b377, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/853b4bf3959e47afbaceeacba0feac26, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d0ffc1c9357d47778b91725005460aba] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=33.9 K 2023-04-20 20:55:57,333 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 834cb95492574e61bcdae4797940b377, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682024154943 2023-04-20 20:55:57,333 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 853b4bf3959e47afbaceeacba0feac26, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682024154983 2023-04-20 20:55:57,334 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d0ffc1c9357d47778b91725005460aba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682024156133 2023-04-20 20:55:57,335 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 2cf1c245a2ed4f8cac38068c65d87775(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:57,335 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:57,335 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=13, startTime=1682024156900; duration=0sec 2023-04-20 20:55:57,335 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:57,339 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#451 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:57,348 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/9ef917660f2e4f5a91fd86ba85c08433 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/9ef917660f2e4f5a91fd86ba85c08433 2023-04-20 20:55:57,354 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into 9ef917660f2e4f5a91fd86ba85c08433(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:57,354 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:57,354 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=13, startTime=1682024156900; duration=0sec 2023-04-20 20:55:57,354 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:57,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024217428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024217428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024217428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024217428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024217429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,443 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/f768d9fafdf4499fa34812541b46cc50 2023-04-20 20:55:57,446 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/736e0fb3e9e14383a36ac00343f51c01 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/736e0fb3e9e14383a36ac00343f51c01 2023-04-20 20:55:57,450 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/736e0fb3e9e14383a36ac00343f51c01, entries=150, sequenceid=382, filesize=11.9 K 2023-04-20 20:55:57,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/d2a02cf52594401d807f5eed5988e91f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d2a02cf52594401d807f5eed5988e91f 2023-04-20 20:55:57,454 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d2a02cf52594401d807f5eed5988e91f, entries=150, sequenceid=382, filesize=11.9 K 2023-04-20 20:55:57,454 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/f768d9fafdf4499fa34812541b46cc50 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f768d9fafdf4499fa34812541b46cc50 2023-04-20 20:55:57,457 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f768d9fafdf4499fa34812541b46cc50, entries=150, sequenceid=382, filesize=11.9 K 2023-04-20 20:55:57,458 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8552cf5289bf62dd18e69f1f78b0b5fd in 462ms, sequenceid=382, compaction requested=false 2023-04-20 20:55:57,458 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:57,458 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,460 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:57,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:57,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ae3898e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:57,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:57,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,468 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:57,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:57,470 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:57,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:57,473 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:57,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:57,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:57,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:57,473 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:57,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:57,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:57,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,476 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:57,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:57,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:57,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:57,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:57,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:57,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:57,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:57,479 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:57,479 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:57,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:57,479 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:57,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:57,580 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:57,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:57,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:57,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:57,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:57,583 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:57,583 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:57,583 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:57,583 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:57,584 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,584 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:57,586 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,586 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:57,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:57,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:57,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:57,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:57,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:57,588 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:57,588 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:57,588 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 20:55:57,588 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:57,589 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:57,589 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:57,589 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:57,589 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:57,589 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:57,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:57,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:57,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024217764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024217765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024217766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024217772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024217772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024217873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024217873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024217873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024217879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:57,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024217879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:57,997 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/b0da2887c58043bca19ccb92508dd0d6 2023-04-20 20:55:58,008 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/b4792c50fc174cda96f82e412d5c6468 2023-04-20 20:55:58,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024218077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024218078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024218079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024218086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024218087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024218385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024218386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024218387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024218396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024218397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,418 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/777f45feb6a84a7dbda387eca49b1034 2023-04-20 20:55:58,422 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/b0da2887c58043bca19ccb92508dd0d6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b0da2887c58043bca19ccb92508dd0d6 2023-04-20 20:55:58,425 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b0da2887c58043bca19ccb92508dd0d6, entries=150, sequenceid=403, filesize=11.9 K 2023-04-20 20:55:58,426 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/b4792c50fc174cda96f82e412d5c6468 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b4792c50fc174cda96f82e412d5c6468 2023-04-20 20:55:58,429 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b4792c50fc174cda96f82e412d5c6468, entries=150, sequenceid=403, filesize=11.9 K 2023-04-20 20:55:58,430 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/777f45feb6a84a7dbda387eca49b1034 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/777f45feb6a84a7dbda387eca49b1034 2023-04-20 20:55:58,433 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/777f45feb6a84a7dbda387eca49b1034, entries=150, sequenceid=403, filesize=11.9 K 2023-04-20 20:55:58,434 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 8552cf5289bf62dd18e69f1f78b0b5fd in 846ms, sequenceid=403, compaction requested=true 2023-04-20 20:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:58,434 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:58,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:58,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:58,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6569bc11[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:58,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:58,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,440 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:58,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:58,441 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:58,443 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:58,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:58,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:58,444 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:58,444 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:58,448 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,448 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,448 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,450 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:58,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:58,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:58,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,453 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:58,453 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:58,553 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:58,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:58,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:58,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:58,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:58,557 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,559 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:58,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:58,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:58,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:58,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:58,562 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:55:58,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:58,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:58,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:58,562 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:58,563 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:58,563 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:58,569 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/8136437e35d2496cb57aa7fc416dcc16 2023-04-20 20:55:58,579 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/089e2c90e15c4e1abe5003c4dd1ee379 2023-04-20 20:55:58,590 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/6087fb919ba2459383f30884106acc3a 2023-04-20 20:55:58,596 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/8136437e35d2496cb57aa7fc416dcc16 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8136437e35d2496cb57aa7fc416dcc16 2023-04-20 20:55:58,599 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8136437e35d2496cb57aa7fc416dcc16, entries=150, sequenceid=418, filesize=11.9 K 2023-04-20 20:55:58,601 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/089e2c90e15c4e1abe5003c4dd1ee379 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/089e2c90e15c4e1abe5003c4dd1ee379 2023-04-20 20:55:58,604 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/089e2c90e15c4e1abe5003c4dd1ee379, entries=150, sequenceid=418, filesize=11.9 K 2023-04-20 20:55:58,604 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/6087fb919ba2459383f30884106acc3a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6087fb919ba2459383f30884106acc3a 2023-04-20 20:55:58,609 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6087fb919ba2459383f30884106acc3a, entries=150, sequenceid=418, filesize=11.9 K 2023-04-20 20:55:58,610 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 48ms, sequenceid=418, compaction requested=true 2023-04-20 20:55:58,610 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:58,610 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:58,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,612 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b61e4b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:58,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:58,614 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,615 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:58,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,616 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:58,617 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:58,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:58,620 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:58,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:58,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:58,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:58,620 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:58,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,622 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,623 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,624 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:58,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,628 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:58,628 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,729 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:58,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:58,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:58,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:58,732 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:58,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:58,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:58,733 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,733 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,735 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:58,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:58,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:58,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:58,737 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,737 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:58,737 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:58,737 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:58,739 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,739 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183cb055[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:58,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:58,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,742 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:58,743 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,743 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:58,745 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:58,748 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:58,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:58,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:58,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:58,748 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:58,750 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,750 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,750 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,752 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:58,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,756 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:58,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,757 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:58,857 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:58,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:58,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:58,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:58,860 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:58,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:58,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:58,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:58,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:58,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:58,863 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:58,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:58,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:58,864 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,867 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:58,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef5e525[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:58,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:58,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,874 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:58,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:58,874 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:58,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:58,876 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:58,878 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:58,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:58,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:58,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:58,878 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:58,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:58,880 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,881 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:58,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:58,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:58,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,889 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,889 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,889 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,889 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:58,889 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:58,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:58,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:55:58,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:55:58,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:58,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:58,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:58,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:58,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:58,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:58,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024218931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024218932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024218936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024218940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:58,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024218940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,989 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:58,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:58,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:58,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:58,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:58,992 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:58,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:58,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:58,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,995 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:58,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:58,995 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,995 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:58,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:58,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:58,997 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:58,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:58,998 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:58,999 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:58,999 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d457de[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:59,001 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:59,001 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,005 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,005 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,005 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:59,007 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:59,009 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:59,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:59,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:59,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:59,009 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:59,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,011 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,013 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:59,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:59,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,016 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:59,016 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:59,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024219041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024219041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024219043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39984 deadline: 1682024219043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024219048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,116 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:59,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:59,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:59,119 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:59,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:59,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:59,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,122 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,122 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,123 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,123 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:59,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:59,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:59,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:59,125 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,125 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:59,125 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:59,125 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,125 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:59,125 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,127 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:59,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23654f09[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:59,129 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:59,129 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,130 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:59,130 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,130 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:59,131 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:59,132 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,132 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:59,132 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,132 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:59,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:59,135 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:59,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:59,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:59,135 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:59,135 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:59,137 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,137 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,137 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,137 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,139 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,141 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:59,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:59,141 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,142 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,142 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,142 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,142 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:59,142 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:59,242 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:59,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:59,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:59,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:59,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:59,246 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:59,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:59,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:59,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,246 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,249 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,249 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,249 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:59,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:59,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:59,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:59,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:59,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024219247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024219248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024219249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024219249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:59,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:59,259 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,259 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:59,259 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:59,259 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:59,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024219254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,262 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,262 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,263 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,263 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e26ed28[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:59,264 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:59,264 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,266 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:59,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:59,266 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:59,267 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,267 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:59,269 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:59,271 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:59,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:59,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:59,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:59,272 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:59,273 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,273 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,273 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,281 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,281 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,281 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:59,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:59,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,281 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,281 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:59,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:59,281 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,282 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:59,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/0e9fe819e1b8409ebf92c0be138389ce 2023-04-20 20:55:59,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/0ea9978d448a4f10bc4cc33f33a85343 2023-04-20 20:55:59,382 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:59,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:59,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:59,385 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:59,385 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,385 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:59,385 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:59,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,387 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,387 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:59,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:59,389 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,389 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:59,389 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:59,389 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:59,391 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f2be704[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:59,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:59,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,394 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:59,395 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,395 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:59,396 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:59,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,397 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:59,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:59,399 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:59,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:59,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:59,399 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:59,399 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:59,400 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,400 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,400 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:59,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:59,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,406 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,406 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,406 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,406 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:59,406 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:59,506 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:59,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:59,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:59,510 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:59,510 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,510 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:59,510 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:59,510 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,510 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,512 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,512 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,512 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,512 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:59,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:59,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:59,513 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:59,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,513 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:59,514 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:59,514 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:59,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:59,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b352b98[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:59,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:59,521 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,523 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:59,523 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,523 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:59,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:59,523 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:59,524 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,524 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:59,527 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:59,527 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:59,527 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:59,527 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:59,527 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:59,528 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,529 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,529 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,530 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:59,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,534 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:59,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:59,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,535 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:59,535 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:59,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024219558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024219558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024219559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024219560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:55:59,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024219563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,635 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:59,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:59,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:59,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:59,638 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:59,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:59,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:59,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:59,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:59,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:59,642 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,642 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:59,643 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:55:59,643 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:55:59,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:55:59,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:55:59,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:55:59,644 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:55:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:55:59,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72213ef0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:55:59,646 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:55:59,646 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,647 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:55:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:55:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:55:59,649 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,649 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:55:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:55:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,652 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:55:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:55:59,654 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:55:59,654 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:55:59,654 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:55:59,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:55:59,655 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:55:59,656 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,656 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:55:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:55:59,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,656 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,656 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:55:59,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:55:59,658 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:55:59,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:55:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:55:59,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:55:59,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:55:59,660 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:55:59,660 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,661 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,661 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,661 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:55:59,661 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,660 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:55:59,661 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:55:59,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:55:59,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/379964679f26416fa270590997c478a2 2023-04-20 20:55:59,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/0e9fe819e1b8409ebf92c0be138389ce as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/0e9fe819e1b8409ebf92c0be138389ce 2023-04-20 20:55:59,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/0e9fe819e1b8409ebf92c0be138389ce, entries=250, sequenceid=432, filesize=16.7 K 2023-04-20 20:55:59,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/0ea9978d448a4f10bc4cc33f33a85343 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0ea9978d448a4f10bc4cc33f33a85343 2023-04-20 20:55:59,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0ea9978d448a4f10bc4cc33f33a85343, entries=100, sequenceid=432, filesize=9.5 K 2023-04-20 20:55:59,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/379964679f26416fa270590997c478a2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/379964679f26416fa270590997c478a2 2023-04-20 20:55:59,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/379964679f26416fa270590997c478a2, entries=100, sequenceid=432, filesize=9.5 K 2023-04-20 20:55:59,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8552cf5289bf62dd18e69f1f78b0b5fd in 848ms, sequenceid=432, compaction requested=true 2023-04-20 20:55:59,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:59,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:59,754 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:55:59,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:59,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:59,754 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:55:59,756 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:55:59,756 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:55:59,756 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,756 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/39554c9e31394031bd312bb466a08cc3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/736e0fb3e9e14383a36ac00343f51c01, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b0da2887c58043bca19ccb92508dd0d6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8136437e35d2496cb57aa7fc416dcc16, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/0e9fe819e1b8409ebf92c0be138389ce] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=64.9 K 2023-04-20 20:55:59,756 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59168 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:55:59,757 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:55:59,757 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,757 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2cf1c245a2ed4f8cac38068c65d87775, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d2a02cf52594401d807f5eed5988e91f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b4792c50fc174cda96f82e412d5c6468, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/089e2c90e15c4e1abe5003c4dd1ee379, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0ea9978d448a4f10bc4cc33f33a85343] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=57.8 K 2023-04-20 20:55:59,757 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 39554c9e31394031bd312bb466a08cc3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682024156133 2023-04-20 20:55:59,757 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2cf1c245a2ed4f8cac38068c65d87775, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682024156133 2023-04-20 20:55:59,757 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 736e0fb3e9e14383a36ac00343f51c01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1682024156459 2023-04-20 20:55:59,757 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d2a02cf52594401d807f5eed5988e91f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1682024156459 2023-04-20 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b0da2887c58043bca19ccb92508dd0d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682024157096 2023-04-20 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b4792c50fc174cda96f82e412d5c6468, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682024157096 2023-04-20 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8136437e35d2496cb57aa7fc416dcc16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682024157764 2023-04-20 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 089e2c90e15c4e1abe5003c4dd1ee379, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682024157764 2023-04-20 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0e9fe819e1b8409ebf92c0be138389ce, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682024158897 2023-04-20 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0ea9978d448a4f10bc4cc33f33a85343, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682024158902 2023-04-20 20:55:59,761 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:55:59,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:55:59,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:55:59,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:55:59,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:55:59,765 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:55:59,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:55:59,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:55:59,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:55:59,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:55:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:55:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,767 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:55:59,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:55:59,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:55:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:55:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:55:59,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:55:59,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:55:59,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:55:59,768 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:55:59,769 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 20:55:59,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:55:59,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:59,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:55:59,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:59,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:55:59,769 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:55:59,771 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#462 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:59,771 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#461 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:59,800 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/b1057428b2ae4a6a8a1e87071015a8ce as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b1057428b2ae4a6a8a1e87071015a8ce 2023-04-20 20:55:59,805 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into b1057428b2ae4a6a8a1e87071015a8ce(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:59,805 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:59,805 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=11, startTime=1682024159754; duration=0sec 2023-04-20 20:55:59,805 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:55:59,806 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:55:59,807 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59168 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:55:59,807 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:55:59,807 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:55:59,807 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/9ef917660f2e4f5a91fd86ba85c08433, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f768d9fafdf4499fa34812541b46cc50, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/777f45feb6a84a7dbda387eca49b1034, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6087fb919ba2459383f30884106acc3a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/379964679f26416fa270590997c478a2] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=57.8 K 2023-04-20 20:55:59,807 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9ef917660f2e4f5a91fd86ba85c08433, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682024156133 2023-04-20 20:55:59,808 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f768d9fafdf4499fa34812541b46cc50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1682024156459 2023-04-20 20:55:59,813 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 777f45feb6a84a7dbda387eca49b1034, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1682024157096 2023-04-20 20:55:59,813 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6087fb919ba2459383f30884106acc3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682024157764 2023-04-20 20:55:59,814 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 379964679f26416fa270590997c478a2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682024158902 2023-04-20 20:55:59,815 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/c78ad043fe1d46ecbda80108f01ea59c 2023-04-20 20:55:59,823 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#464 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:55:59,842 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/e13c43c565db460bb3b910a053f98462 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e13c43c565db460bb3b910a053f98462 2023-04-20 20:55:59,848 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into e13c43c565db460bb3b910a053f98462(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:55:59,849 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:55:59,849 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=11, startTime=1682024159754; duration=0sec 2023-04-20 20:55:59,849 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:55:59,850 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/55d1404fd6d9423ab6d74d65e5734517 2023-04-20 20:56:00,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:00,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:00,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024220080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024220080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024220081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024220081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024220081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,189 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/5fcb26efda934ff68fa7514552f6157e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/5fcb26efda934ff68fa7514552f6157e 2023-04-20 20:56:00,192 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 5fcb26efda934ff68fa7514552f6157e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:00,192 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:00,192 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=11, startTime=1682024159754; duration=0sec 2023-04-20 20:56:00,193 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:00,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024220193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024220203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024220204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024220204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024220204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,271 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/07725cd7eb0c4e229a0942d590a4425e 2023-04-20 20:56:00,275 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/c78ad043fe1d46ecbda80108f01ea59c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/c78ad043fe1d46ecbda80108f01ea59c 2023-04-20 20:56:00,279 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/c78ad043fe1d46ecbda80108f01ea59c, entries=150, sequenceid=452, filesize=11.9 K 2023-04-20 20:56:00,279 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/55d1404fd6d9423ab6d74d65e5734517 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/55d1404fd6d9423ab6d74d65e5734517 2023-04-20 20:56:00,283 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/55d1404fd6d9423ab6d74d65e5734517, entries=150, sequenceid=452, filesize=11.9 K 2023-04-20 20:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/07725cd7eb0c4e229a0942d590a4425e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/07725cd7eb0c4e229a0942d590a4425e 2023-04-20 20:56:00,287 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/07725cd7eb0c4e229a0942d590a4425e, entries=150, sequenceid=452, filesize=11.9 K 2023-04-20 20:56:00,287 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 8552cf5289bf62dd18e69f1f78b0b5fd in 518ms, sequenceid=452, compaction requested=false 2023-04-20 20:56:00,288 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:00,288 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:00,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,293 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,293 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,293 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b87d231[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:00,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:00,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:00,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:00,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:00,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:00,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:00,303 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:00,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:00,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:00,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:00,304 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:00,306 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,306 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,306 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,307 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,311 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,311 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,311 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:00,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:00,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,312 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:00,312 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:00,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:00,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 20:56:00,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:00,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:00,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:00,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:00,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:00,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:00,412 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:00,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:00,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:00,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:00,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:00,415 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:00,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:00,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:00,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,417 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,418 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,418 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:00,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/87950f5b8a9c4c74b68dbbb62b5dae10 2023-04-20 20:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:00,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:00,420 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,420 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:00,420 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:00,420 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:00,422 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bc41582[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:00,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:00,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,427 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:00,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:00,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,429 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:00,432 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:00,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:00,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:00,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:00,432 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:00,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024220430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024220431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024220431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024220432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,440 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,442 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:00,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:00,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,443 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:00,443 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:00,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024220439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,543 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:00,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:00,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024220540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024220540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024220540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024220540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:00,546 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:00,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:00,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:00,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024220546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,553 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,553 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:00,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:00,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:00,555 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,555 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:00,556 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:00,556 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:00,557 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,558 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@197f8e89[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:00,560 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:00,560 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:00,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:00,565 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:00,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:00,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:00,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:00,566 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:00,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,572 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,572 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:00,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:00,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,573 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:00,573 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:00,673 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:00,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:00,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:00,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:00,676 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:00,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:00,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:00,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,678 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:00,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:00,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,681 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:00,683 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d2e5e8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:00,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:00,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:00,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:00,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:00,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,688 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:00,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:00,691 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:00,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:00,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:00,692 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:00,692 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:00,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,694 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,695 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:00,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:00,697 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,697 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,698 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:00,698 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:00,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:00,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024220745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024220746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024220747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024220748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024220749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,798 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:00,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:00,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:00,803 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:00,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:00,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:00,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,806 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:00,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:00,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:00,807 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:00,807 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,807 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:00,808 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,808 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:00,808 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:00,808 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:00,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:00,810 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,810 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,810 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,810 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4761a5d9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:00,812 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:00,812 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:00,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:00,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:00,815 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:00,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,815 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,815 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:00,816 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:00,816 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:00,816 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:00,816 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:00,816 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:00,816 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:00,816 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:00,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,819 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,825 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,824 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,825 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:00,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,826 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:00,826 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:00,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/b7c473f76d18449cb3a73a8288f2b8d3 2023-04-20 20:56:00,926 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:00,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:00,929 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:00,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:00,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:00,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:00,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:00,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:00,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:00,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:00,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:00,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:00,933 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,933 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:00,933 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:00,933 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:00,935 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:00,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce2d0b2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:00,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:00,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,939 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:00,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:00,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:00,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:00,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,941 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:00,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:00,943 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:00,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:00,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:00,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:00,944 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:00,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:00,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:00,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:00,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,946 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,948 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:00,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,951 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:00,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:00,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:00,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:00,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:00,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:00,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:00,952 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:00,952 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:01,052 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:01,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:01,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024221049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:01,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:01,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:01,055 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:01,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:01,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:01,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024221051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,058 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,058 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:01,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:01,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:01,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:01,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:01,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:01,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,062 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,063 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024221054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024221056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024221056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:01,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@490c1861[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:01,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:01,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:01,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:01,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:01,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:01,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:01,070 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:01,070 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:01,070 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:01,070 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:01,070 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:01,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,082 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:01,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:01,085 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,085 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,085 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:01,085 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:01,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:01,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,185 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:01,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:01,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:01,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:01,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:01,188 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:01,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:01,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:01,189 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,189 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,190 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:01,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:01,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:01,193 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,193 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:01,193 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:01,193 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:01,193 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:01,195 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,195 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:01,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c04590d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:01,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:01,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,199 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:01,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:01,199 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,199 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:01,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:01,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:01,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:01,205 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:01,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:01,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:01,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:01,206 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:01,207 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,207 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,207 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,210 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,212 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:01,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:01,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,213 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:01,213 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:01,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/4338575858e54808a61bfc48fc6a94f7 2023-04-20 20:56:01,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/87950f5b8a9c4c74b68dbbb62b5dae10 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/87950f5b8a9c4c74b68dbbb62b5dae10 2023-04-20 20:56:01,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/87950f5b8a9c4c74b68dbbb62b5dae10, entries=150, sequenceid=470, filesize=11.9 K 2023-04-20 20:56:01,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/b7c473f76d18449cb3a73a8288f2b8d3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b7c473f76d18449cb3a73a8288f2b8d3 2023-04-20 20:56:01,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b7c473f76d18449cb3a73a8288f2b8d3, entries=150, sequenceid=470, filesize=11.9 K 2023-04-20 20:56:01,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/4338575858e54808a61bfc48fc6a94f7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/4338575858e54808a61bfc48fc6a94f7 2023-04-20 20:56:01,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/4338575858e54808a61bfc48fc6a94f7, entries=150, sequenceid=470, filesize=11.9 K 2023-04-20 20:56:01,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8552cf5289bf62dd18e69f1f78b0b5fd in 873ms, sequenceid=470, compaction requested=true 2023-04-20 20:56:01,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:01,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:01,283 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:01,283 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:01,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:01,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:01,283 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:01,284 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:56:01,284 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:01,284 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,284 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:56:01,284 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b1057428b2ae4a6a8a1e87071015a8ce, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/c78ad043fe1d46ecbda80108f01ea59c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/87950f5b8a9c4c74b68dbbb62b5dae10] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.5 K 2023-04-20 20:56:01,284 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,284 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/5fcb26efda934ff68fa7514552f6157e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/55d1404fd6d9423ab6d74d65e5734517, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b7c473f76d18449cb3a73a8288f2b8d3] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.5 K 2023-04-20 20:56:01,285 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b1057428b2ae4a6a8a1e87071015a8ce, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682024157764 2023-04-20 20:56:01,286 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5fcb26efda934ff68fa7514552f6157e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682024157764 2023-04-20 20:56:01,286 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting c78ad043fe1d46ecbda80108f01ea59c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682024158930 2023-04-20 20:56:01,287 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 55d1404fd6d9423ab6d74d65e5734517, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682024158930 2023-04-20 20:56:01,287 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 87950f5b8a9c4c74b68dbbb62b5dae10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682024160080 2023-04-20 20:56:01,287 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b7c473f76d18449cb3a73a8288f2b8d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682024160080 2023-04-20 20:56:01,292 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#470 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:01,294 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:01,313 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:01,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:01,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:01,315 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:01,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:01,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:01,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,318 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,318 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:01,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:01,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:01,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:01,320 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:01,320 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,320 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:01,320 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:56:01,320 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:01,320 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:01,320 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:01,320 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:01,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:01,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:01,333 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/b4d696a4062c449eb8028bff4589c67f 2023-04-20 20:56:01,347 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/68a4f6fe6439407489a3ed1149d2d9db 2023-04-20 20:56:01,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:01,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:01,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024221576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024221577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024221581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024221581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024221584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024221685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024221687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024221688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024221689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024221689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,704 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/a7d0d3d294a14a93b01b4dc51589330e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a7d0d3d294a14a93b01b4dc51589330e 2023-04-20 20:56:01,707 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/682187ec20bf41f88b0c943d0afd56d9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/682187ec20bf41f88b0c943d0afd56d9 2023-04-20 20:56:01,710 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into a7d0d3d294a14a93b01b4dc51589330e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:01,711 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:01,711 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=13, startTime=1682024161283; duration=0sec 2023-04-20 20:56:01,711 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:01,711 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:01,712 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 682187ec20bf41f88b0c943d0afd56d9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:01,712 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:01,712 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=13, startTime=1682024161283; duration=0sec 2023-04-20 20:56:01,713 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:01,713 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:01,713 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:56:01,713 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,713 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e13c43c565db460bb3b910a053f98462, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/07725cd7eb0c4e229a0942d590a4425e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/4338575858e54808a61bfc48fc6a94f7] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.5 K 2023-04-20 20:56:01,714 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e13c43c565db460bb3b910a053f98462, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682024157764 2023-04-20 20:56:01,714 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 07725cd7eb0c4e229a0942d590a4425e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682024158930 2023-04-20 20:56:01,715 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 4338575858e54808a61bfc48fc6a94f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682024160080 2023-04-20 20:56:01,722 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:01,735 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/b20e295d3c17432fb3702a810dcd00a0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b20e295d3c17432fb3702a810dcd00a0 2023-04-20 20:56:01,741 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into b20e295d3c17432fb3702a810dcd00a0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:01,741 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:01,741 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=13, startTime=1682024161283; duration=0sec 2023-04-20 20:56:01,741 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:01,756 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/70ded06e777e4a2ba65cfa95a8073d7f 2023-04-20 20:56:01,761 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/b4d696a4062c449eb8028bff4589c67f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b4d696a4062c449eb8028bff4589c67f 2023-04-20 20:56:01,764 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b4d696a4062c449eb8028bff4589c67f, entries=150, sequenceid=487, filesize=11.9 K 2023-04-20 20:56:01,765 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/68a4f6fe6439407489a3ed1149d2d9db as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/68a4f6fe6439407489a3ed1149d2d9db 2023-04-20 20:56:01,770 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/68a4f6fe6439407489a3ed1149d2d9db, entries=150, sequenceid=487, filesize=11.9 K 2023-04-20 20:56:01,771 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/70ded06e777e4a2ba65cfa95a8073d7f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70ded06e777e4a2ba65cfa95a8073d7f 2023-04-20 20:56:01,778 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70ded06e777e4a2ba65cfa95a8073d7f, entries=150, sequenceid=487, filesize=11.9 K 2023-04-20 20:56:01,779 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8552cf5289bf62dd18e69f1f78b0b5fd in 458ms, sequenceid=487, compaction requested=false 2023-04-20 20:56:01,779 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:01,779 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:01,782 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76817c39[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:01,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:01,788 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:01,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:01,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:01,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:01,791 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:01,795 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:01,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:01,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:01,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:01,795 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:01,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,798 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,798 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,801 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:01,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,804 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:01,804 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,804 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:01,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 20:56:01,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:01,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:01,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:01,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:01,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:01,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:01,904 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:01,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:01,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:01,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:01,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:01,907 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:01,907 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,907 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:01,907 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:01,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,909 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,910 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,910 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:01,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:01,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:01,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:01,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:01,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:01,911 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:01,912 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:01,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,913 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cdc451b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:01,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:01,915 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024221909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024221910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024221911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024221915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,918 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,918 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:01,918 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:01,918 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:01,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:01,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:01,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:01,922 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:01,922 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:01,922 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:01,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:01,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024221916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:01,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:01,925 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,926 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:01,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:01,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:01,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:01,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:01,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:01,932 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:01,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:01,932 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:02,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024222017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024222018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024222018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024222019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024222024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,033 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:02,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:02,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:02,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:02,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:02,037 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:02,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:02,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:02,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,040 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,040 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:02,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:02,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:02,042 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,042 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:02,042 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:02,042 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:02,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,045 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ba83f1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:02,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:02,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,049 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:02,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:02,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:02,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:02,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:02,055 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:02,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:02,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:02,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:02,055 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:02,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,060 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,062 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:02,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,062 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:02,063 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:02,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:02,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,163 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:02,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:02,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:02,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:02,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:02,166 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:02,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,167 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:02,167 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:02,167 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,167 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,169 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,169 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,169 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:02,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:02,171 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,171 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:02,171 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:02,171 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:02,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:02,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,173 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b68727a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:02,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:02,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:02,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:02,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:02,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,180 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:02,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:02,183 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:02,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:02,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:02,183 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:02,183 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:02,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,191 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,191 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:02,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,191 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,191 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:02,192 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:02,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,191 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024222222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024222222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024222223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024222223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024222228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,292 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:02,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:02,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:02,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:02,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:02,295 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:02,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:02,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:02,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:02,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:02,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:02,298 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,298 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:02,298 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:02,298 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:02,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:02,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5a99a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:02,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:02,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/fc90e4d0559442c2a481eb443b966b19 2023-04-20 20:56:02,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:02,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:02,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:02,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:02,306 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:02,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:02,308 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:02,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:02,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:02,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:02,309 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:02,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,311 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/f1ef9033cd8447c591a4cda71dc98107 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,317 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:02,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:02,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,318 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:02,318 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:02,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:02,418 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:02,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:02,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:02,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:02,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:02,422 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:02,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:02,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:02,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:02,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:02,426 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,426 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:02,426 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:02,426 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,427 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:02,427 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:02,429 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282fd0c2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:02,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:02,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:02,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,437 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:02,440 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:02,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:02,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:02,441 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:02,441 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:02,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,445 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:02,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:02,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:02,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,452 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:02,451 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,452 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:02,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024222526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024222527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024222528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024222529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:02,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024222533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,552 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:02,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:02,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:02,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:02,555 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:02,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:02,555 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:02,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,557 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,557 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:02,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:02,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:02,559 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,559 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:02,559 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:02,559 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:02,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@193b6b3d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:02,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:02,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:02,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:02,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:02,564 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,564 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:02,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,566 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:02,566 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,566 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:02,569 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:02,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:02,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:02,569 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:02,569 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:02,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,570 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,573 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,575 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:02,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:02,575 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,576 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:02,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:02,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,576 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:02,676 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:02,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:02,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:02,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:02,680 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:02,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:02,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:02,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:02,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21f831c2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:02,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:02,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,692 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:02,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:02,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,694 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:02,696 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:02,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:02,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:02,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:02,696 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:02,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:02,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:02,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,701 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:02,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:02,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:02,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:02,704 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:02,704 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:02,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:02,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/2fa5725e699f430fa238825665a15613 2023-04-20 20:56:02,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/fc90e4d0559442c2a481eb443b966b19 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/fc90e4d0559442c2a481eb443b966b19 2023-04-20 20:56:02,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/fc90e4d0559442c2a481eb443b966b19, entries=200, sequenceid=506, filesize=14.3 K 2023-04-20 20:56:02,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/f1ef9033cd8447c591a4cda71dc98107 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/f1ef9033cd8447c591a4cda71dc98107 2023-04-20 20:56:02,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/f1ef9033cd8447c591a4cda71dc98107, entries=150, sequenceid=506, filesize=11.9 K 2023-04-20 20:56:02,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/2fa5725e699f430fa238825665a15613 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2fa5725e699f430fa238825665a15613 2023-04-20 20:56:02,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2fa5725e699f430fa238825665a15613, entries=150, sequenceid=506, filesize=11.9 K 2023-04-20 20:56:02,742 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8552cf5289bf62dd18e69f1f78b0b5fd in 850ms, sequenceid=506, compaction requested=true 2023-04-20 20:56:02,742 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:02,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:02,742 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:02,742 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:02,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:02,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:02,743 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:02,743 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:02,743 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:56:02,743 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:56:02,743 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,743 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,743 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a7d0d3d294a14a93b01b4dc51589330e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b4d696a4062c449eb8028bff4589c67f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/fc90e4d0559442c2a481eb443b966b19] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=39.0 K 2023-04-20 20:56:02,743 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/682187ec20bf41f88b0c943d0afd56d9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/68a4f6fe6439407489a3ed1149d2d9db, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/f1ef9033cd8447c591a4cda71dc98107] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.6 K 2023-04-20 20:56:02,744 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting a7d0d3d294a14a93b01b4dc51589330e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682024160080 2023-04-20 20:56:02,744 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 682187ec20bf41f88b0c943d0afd56d9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682024160080 2023-04-20 20:56:02,744 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b4d696a4062c449eb8028bff4589c67f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682024160427 2023-04-20 20:56:02,744 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 68a4f6fe6439407489a3ed1149d2d9db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682024160427 2023-04-20 20:56:02,745 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting fc90e4d0559442c2a481eb443b966b19, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682024161575 2023-04-20 20:56:02,745 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f1ef9033cd8447c591a4cda71dc98107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682024161575 2023-04-20 20:56:02,755 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#479 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:02,756 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#480 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:02,773 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/44a7569825b448b486399335a4c8536f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/44a7569825b448b486399335a4c8536f 2023-04-20 20:56:02,778 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 44a7569825b448b486399335a4c8536f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:02,778 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:02,778 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=13, startTime=1682024162742; duration=0sec 2023-04-20 20:56:02,778 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:02,778 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:02,779 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:02,780 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:56:02,780 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,780 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b20e295d3c17432fb3702a810dcd00a0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70ded06e777e4a2ba65cfa95a8073d7f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2fa5725e699f430fa238825665a15613] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.6 K 2023-04-20 20:56:02,780 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b20e295d3c17432fb3702a810dcd00a0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682024160080 2023-04-20 20:56:02,781 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 70ded06e777e4a2ba65cfa95a8073d7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682024160427 2023-04-20 20:56:02,781 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2fa5725e699f430fa238825665a15613, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682024161575 2023-04-20 20:56:02,789 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:02,801 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/7115d07c802e4f098e6ccd16f1f3d8ae as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7115d07c802e4f098e6ccd16f1f3d8ae 2023-04-20 20:56:02,804 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:02,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:02,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:02,807 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into 7115d07c802e4f098e6ccd16f1f3d8ae(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:02,807 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:02,807 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=13, startTime=1682024162742; duration=0sec 2023-04-20 20:56:02,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:02,807 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:02,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:02,808 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:02,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:02,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:02,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,809 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,809 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:02,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:02,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:02,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:02,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:02,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:02,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:02,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:02,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:02,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:02,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:02,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:02,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:02,811 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 20:56:02,812 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:02,812 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:02,812 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:02,812 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:02,812 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:02,812 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:02,832 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/01d379cc26884e00a621080bf5bcadf6 2023-04-20 20:56:03,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:03,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:03,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024223048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024223050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024223051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024223052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024223052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024223153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024223159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,164 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/5e4e803bba114fb19d000aa882e269db as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5e4e803bba114fb19d000aa882e269db 2023-04-20 20:56:03,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024223159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024223159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024223159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,168 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 5e4e803bba114fb19d000aa882e269db(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:03,168 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:03,168 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=13, startTime=1682024162742; duration=0sec 2023-04-20 20:56:03,169 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:03,247 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/0b93aec68a664718add8d08085de0d49 2023-04-20 20:56:03,258 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d536cb24cda64e789957f98f5de5b38e 2023-04-20 20:56:03,263 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/01d379cc26884e00a621080bf5bcadf6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/01d379cc26884e00a621080bf5bcadf6 2023-04-20 20:56:03,269 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/01d379cc26884e00a621080bf5bcadf6, entries=150, sequenceid=524, filesize=11.9 K 2023-04-20 20:56:03,269 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/0b93aec68a664718add8d08085de0d49 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0b93aec68a664718add8d08085de0d49 2023-04-20 20:56:03,272 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0b93aec68a664718add8d08085de0d49, entries=150, sequenceid=524, filesize=11.9 K 2023-04-20 20:56:03,274 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d536cb24cda64e789957f98f5de5b38e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d536cb24cda64e789957f98f5de5b38e 2023-04-20 20:56:03,277 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d536cb24cda64e789957f98f5de5b38e, entries=150, sequenceid=524, filesize=11.9 K 2023-04-20 20:56:03,278 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8552cf5289bf62dd18e69f1f78b0b5fd in 467ms, sequenceid=524, compaction requested=false 2023-04-20 20:56:03,278 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:03,278 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:03,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a6af51e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:03,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:03,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,285 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:03,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:03,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:03,290 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:03,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:03,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:03,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:03,290 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:03,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,299 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:03,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:03,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,300 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:03,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,300 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:03,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:03,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:56:03,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:03,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:03,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:03,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:03,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:03,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:03,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/ed8ab8259ca0462aa682ea22ed5be2b0 2023-04-20 20:56:03,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/d47c3255f52441a0997ff51b90f1a61e 2023-04-20 20:56:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024223378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024223379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024223379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024223379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024223382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,400 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:03,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:03,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:03,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:03,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:03,403 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:03,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:03,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:03,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,405 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:03,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:03,408 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c159d51[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:03,411 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:03,411 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,412 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:03,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:03,415 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:03,417 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:03,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:03,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:03,418 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:03,418 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:03,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,421 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:03,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,425 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:03,425 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:03,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:03,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024223486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024223486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024223488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024223488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024223488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,526 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:03,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:03,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:03,529 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:03,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:03,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:03,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:03,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:03,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:03,533 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,533 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:03,533 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:03,533 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:03,536 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,536 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,536 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,536 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb5676d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:03,537 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:03,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:03,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:03,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:03,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:03,541 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:03,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:03,543 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:03,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:03,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:03,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:03,544 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:03,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,550 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,550 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,550 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:03,550 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,551 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,550 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:03,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:03,551 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,551 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,551 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,551 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,551 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,551 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:03,651 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:03,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:03,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:03,655 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:03,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:03,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:03,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,657 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,657 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:03,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:03,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:03,659 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,659 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:03,659 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:03,659 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:03,662 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:03,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e233358[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:03,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:03,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,669 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:03,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:03,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:03,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:03,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:03,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:03,674 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:03,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,676 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:03,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:03,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:03,680 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:03,680 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024223691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024223692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024223692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024223696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:03,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024223696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,780 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:03,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:03,783 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:03,783 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,783 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:03,783 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:03,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,786 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:03,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:03,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:03,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:03,790 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/0117eb5432b447b083a0b10dd6e2024d 2023-04-20 20:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:03,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23bceb87[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:03,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:03,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,793 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:03,793 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,793 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:03,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:03,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/ed8ab8259ca0462aa682ea22ed5be2b0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ed8ab8259ca0462aa682ea22ed5be2b0 2023-04-20 20:56:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:03,797 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:03,797 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:03,797 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:03,797 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:03,797 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:03,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ed8ab8259ca0462aa682ea22ed5be2b0, entries=200, sequenceid=542, filesize=14.3 K 2023-04-20 20:56:03,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/d47c3255f52441a0997ff51b90f1a61e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d47c3255f52441a0997ff51b90f1a61e 2023-04-20 20:56:03,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,802 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d47c3255f52441a0997ff51b90f1a61e, entries=150, sequenceid=542, filesize=11.9 K 2023-04-20 20:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/0117eb5432b447b083a0b10dd6e2024d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/0117eb5432b447b083a0b10dd6e2024d 2023-04-20 20:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,804 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/0117eb5432b447b083a0b10dd6e2024d, entries=150, sequenceid=542, filesize=11.9 K 2023-04-20 20:56:03,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8552cf5289bf62dd18e69f1f78b0b5fd in 445ms, sequenceid=542, compaction requested=true 2023-04-20 20:56:03,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:03,807 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,807 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:03,807 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:03,807 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:03,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:03,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:03,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:03,808 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:03,807 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,807 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,807 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,808 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:03,808 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:03,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,809 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:03,809 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:56:03,809 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:03,809 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,809 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:56:03,809 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5e4e803bba114fb19d000aa882e269db, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/01d379cc26884e00a621080bf5bcadf6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ed8ab8259ca0462aa682ea22ed5be2b0] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=39.1 K 2023-04-20 20:56:03,809 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,809 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/44a7569825b448b486399335a4c8536f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0b93aec68a664718add8d08085de0d49, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d47c3255f52441a0997ff51b90f1a61e] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.7 K 2023-04-20 20:56:03,810 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 5e4e803bba114fb19d000aa882e269db, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682024161575 2023-04-20 20:56:03,810 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 44a7569825b448b486399335a4c8536f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682024161575 2023-04-20 20:56:03,810 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 01d379cc26884e00a621080bf5bcadf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1682024161910 2023-04-20 20:56:03,810 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0b93aec68a664718add8d08085de0d49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1682024161910 2023-04-20 20:56:03,811 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ed8ab8259ca0462aa682ea22ed5be2b0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682024163042 2023-04-20 20:56:03,811 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d47c3255f52441a0997ff51b90f1a61e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682024163042 2023-04-20 20:56:03,817 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:03,818 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#489 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:03,833 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/09fddda0bb9c4d6e84990ae57c3ad80d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/09fddda0bb9c4d6e84990ae57c3ad80d 2023-04-20 20:56:03,835 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/dcd775c97dbd465d91a079c18aecf31e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/dcd775c97dbd465d91a079c18aecf31e 2023-04-20 20:56:03,838 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 09fddda0bb9c4d6e84990ae57c3ad80d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:03,838 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:03,838 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=13, startTime=1682024163807; duration=0sec 2023-04-20 20:56:03,839 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:03,839 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:03,839 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into dcd775c97dbd465d91a079c18aecf31e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:03,839 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:03,839 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:03,840 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:56:03,840 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=13, startTime=1682024163807; duration=0sec 2023-04-20 20:56:03,840 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,840 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:03,840 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7115d07c802e4f098e6ccd16f1f3d8ae, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d536cb24cda64e789957f98f5de5b38e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/0117eb5432b447b083a0b10dd6e2024d] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.7 K 2023-04-20 20:56:03,840 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 7115d07c802e4f098e6ccd16f1f3d8ae, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1682024161575 2023-04-20 20:56:03,840 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d536cb24cda64e789957f98f5de5b38e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1682024161910 2023-04-20 20:56:03,841 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0117eb5432b447b083a0b10dd6e2024d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682024163042 2023-04-20 20:56:03,849 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#490 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:03,908 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:03,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:03,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:03,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:03,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:03,912 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:03,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:03,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:03,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,912 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,913 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:03,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:03,914 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,914 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:03,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:03,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,915 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:03,916 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:56:03,916 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:03,916 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:03,916 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:03,916 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:03,916 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:03,916 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:03,930 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/08939d4fd9aa49db9086bf21f65091d1 2023-04-20 20:56:03,947 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/0134cede14a84bfe90455f20d5354437 2023-04-20 20:56:03,966 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/3794802455114bba9906f6009009fc05 2023-04-20 20:56:03,970 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/08939d4fd9aa49db9086bf21f65091d1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/08939d4fd9aa49db9086bf21f65091d1 2023-04-20 20:56:03,973 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/08939d4fd9aa49db9086bf21f65091d1, entries=150, sequenceid=559, filesize=11.9 K 2023-04-20 20:56:03,974 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/0134cede14a84bfe90455f20d5354437 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0134cede14a84bfe90455f20d5354437 2023-04-20 20:56:03,977 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0134cede14a84bfe90455f20d5354437, entries=150, sequenceid=559, filesize=11.9 K 2023-04-20 20:56:03,978 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/3794802455114bba9906f6009009fc05 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3794802455114bba9906f6009009fc05 2023-04-20 20:56:03,981 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3794802455114bba9906f6009009fc05, entries=150, sequenceid=559, filesize=11.9 K 2023-04-20 20:56:03,982 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 67ms, sequenceid=559, compaction requested=false 2023-04-20 20:56:03,982 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:03,982 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:03,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:03,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:03,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:03,982 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,984 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,984 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:03,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@533315b4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:03,986 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:03,986 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:03,988 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,988 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:03,989 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:03,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:03,992 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:03,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:03,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:03,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:03,992 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:03,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:03,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,995 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:03,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:03,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:03,996 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:03,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,000 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,000 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,001 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,001 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,001 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:04,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,002 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:04,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:04,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:04,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:04,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:04,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:04,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:04,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:04,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:04,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024164019, max timestamp=1682024164034Num uniques 859; 2023-04-20 20:56:04,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024164019, max timestamp=1682024164034Num uniques 859; 2023-04-20 20:56:04,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024164019, max timestamp=1682024164034Num uniques 859; 2023-04-20 20:56:04,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:04,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:04,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:04,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024224060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024224060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024224069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024224070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024224071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,101 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:04,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:04,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:04,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:04,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:04,104 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:04,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:04,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:04,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,105 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,107 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,107 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,107 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:04,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:04,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:04,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:04,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:04,109 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,109 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:04,109 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:04,109 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:04,110 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:04,111 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:04,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:04,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66088eb0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:04,112 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:04,113 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,114 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,114 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:04,114 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:04,114 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:04,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:04,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:04,119 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,119 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:04,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:04,121 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:04,121 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:04,121 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:04,121 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:04,121 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:04,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,123 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,125 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,126 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,128 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:04,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,128 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,128 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:04,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024224172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024224172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024224183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024224185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024224185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,229 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:04,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:04,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:04,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:04,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:04,232 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:04,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:04,232 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:04,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,234 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,235 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,235 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:04,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:04,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:04,236 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,236 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:04,236 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:04,236 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:04,237 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:04,237 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,237 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:04,237 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:04,239 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,239 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,240 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,240 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@156c8da[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:04,241 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:04,242 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,243 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:04,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:04,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:04,243 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,243 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:04,245 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,245 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:04,245 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,245 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:04,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:04,247 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:04,248 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:04,248 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:04,248 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:04,248 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:04,249 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,249 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,249 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,250 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,251 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,254 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,254 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,255 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,255 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,255 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:04,262 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/fffe3680b98d4eaea003f59094976c50 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/fffe3680b98d4eaea003f59094976c50 2023-04-20 20:56:04,267 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into fffe3680b98d4eaea003f59094976c50(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:04,267 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:04,267 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=13, startTime=1682024163808; duration=0sec 2023-04-20 20:56:04,267 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:04,355 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:04,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:04,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:04,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:04,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:04,359 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:04,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:04,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:04,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,361 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,361 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,362 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:04,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:04,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:04,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:04,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:04,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:04,363 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,363 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:04,363 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:04,363 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,363 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:04,365 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,367 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:04,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:04,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f2eba4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:04,370 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:04,370 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,371 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,371 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:04,371 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:04,371 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,377 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:04,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:04,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:04,377 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:04,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:04,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:04,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:04,377 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:04,379 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,379 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,379 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,380 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:04,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,384 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,384 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,384 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:04,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024224381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024224381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024224391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024224391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024224391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/ee415c946ab145299215622f3d3f3926 2023-04-20 20:56:04,484 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:04,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:04,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:04,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:04,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:04,487 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:04,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:04,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:04,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,489 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,489 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,490 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,490 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:04,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:04,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:04,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:04,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:04,492 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,492 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:04,492 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:04,492 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:04,494 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,494 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,495 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,495 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e27d81b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:04,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:04,496 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,498 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:04,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,498 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:04,500 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:04,500 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,500 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,500 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:04,502 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:04,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:04,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:04,502 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:04,502 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:04,503 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,503 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,503 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,504 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,512 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,512 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:04,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,513 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,513 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:04,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,513 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,613 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:04,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:04,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:04,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:04,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:04,617 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:04,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:04,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:04,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,618 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,618 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:04,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:04,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:04,621 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:04,621 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:04,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:04,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:04,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:04,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:04,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c78ba61[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:04,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:04,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,627 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:04,627 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,627 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:04,629 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:04,632 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:04,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:04,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:04,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:04,632 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:04,634 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,634 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,634 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,634 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,637 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,640 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,641 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,641 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:04,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:04,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024224691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024224691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024224693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024224698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:04,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024224699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,741 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:04,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:04,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:04,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:04,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:04,744 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,745 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,746 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,747 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,747 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:04,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:04,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:04,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:04,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:04,748 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:04,749 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,749 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:04,749 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:04,749 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:04,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:04,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:04,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b5fa72c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:04,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:04,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,755 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:04,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:04,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:04,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:04,757 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,757 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:04,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:04,760 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:04,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:04,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:04,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:04,760 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:04,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,763 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:04,768 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,768 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:04,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/7f165168bda446ffb699fd8866c5c6cf 2023-04-20 20:56:04,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/88c189b56fae4ae48b6bb2966c0bd880 2023-04-20 20:56:04,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/ee415c946ab145299215622f3d3f3926 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ee415c946ab145299215622f3d3f3926 2023-04-20 20:56:04,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ee415c946ab145299215622f3d3f3926, entries=250, sequenceid=570, filesize=16.7 K 2023-04-20 20:56:04,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/7f165168bda446ffb699fd8866c5c6cf as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/7f165168bda446ffb699fd8866c5c6cf 2023-04-20 20:56:04,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/7f165168bda446ffb699fd8866c5c6cf, entries=150, sequenceid=570, filesize=11.9 K 2023-04-20 20:56:04,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/88c189b56fae4ae48b6bb2966c0bd880 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/88c189b56fae4ae48b6bb2966c0bd880 2023-04-20 20:56:04,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/88c189b56fae4ae48b6bb2966c0bd880, entries=150, sequenceid=570, filesize=11.9 K 2023-04-20 20:56:04,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8552cf5289bf62dd18e69f1f78b0b5fd in 845ms, sequenceid=570, compaction requested=true 2023-04-20 20:56:04,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:04,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:04,849 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:04,849 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:04,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:04,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:04,850 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42596 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:04,850 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:04,850 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:56:04,850 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:56:04,850 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,850 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,850 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/09fddda0bb9c4d6e84990ae57c3ad80d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/08939d4fd9aa49db9086bf21f65091d1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ee415c946ab145299215622f3d3f3926] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=41.6 K 2023-04-20 20:56:04,851 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/dcd775c97dbd465d91a079c18aecf31e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0134cede14a84bfe90455f20d5354437, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/7f165168bda446ffb699fd8866c5c6cf] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.8 K 2023-04-20 20:56:04,851 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 09fddda0bb9c4d6e84990ae57c3ad80d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682024163042 2023-04-20 20:56:04,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting dcd775c97dbd465d91a079c18aecf31e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682024163042 2023-04-20 20:56:04,851 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 08939d4fd9aa49db9086bf21f65091d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1682024163376 2023-04-20 20:56:04,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0134cede14a84bfe90455f20d5354437, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1682024163376 2023-04-20 20:56:04,851 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting ee415c946ab145299215622f3d3f3926, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682024163997 2023-04-20 20:56:04,851 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7f165168bda446ffb699fd8866c5c6cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682024163997 2023-04-20 20:56:04,860 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:04,860 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:04,869 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:04,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:04,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:04,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:04,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:04,872 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:04,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:04,872 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:04,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,874 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,875 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:04,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:04,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:04,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:04,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:04,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:04,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:04,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:04,880 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:04,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:04,880 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-20 20:56:04,880 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:04,882 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/915f511d5dcc49a68025cd1c54ac80d0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/915f511d5dcc49a68025cd1c54ac80d0 2023-04-20 20:56:04,886 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/9bce135cf7af4a1eb0191ad6b827bcd6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9bce135cf7af4a1eb0191ad6b827bcd6 2023-04-20 20:56:04,889 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 915f511d5dcc49a68025cd1c54ac80d0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:04,890 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:04,890 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=13, startTime=1682024164849; duration=0sec 2023-04-20 20:56:04,890 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:04,890 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:04,891 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:04,891 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/70727f565cec4198b7dc82173b7b47fc 2023-04-20 20:56:04,891 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:56:04,891 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,891 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/fffe3680b98d4eaea003f59094976c50, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3794802455114bba9906f6009009fc05, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/88c189b56fae4ae48b6bb2966c0bd880] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=36.8 K 2023-04-20 20:56:04,892 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting fffe3680b98d4eaea003f59094976c50, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682024163042 2023-04-20 20:56:04,892 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 3794802455114bba9906f6009009fc05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1682024163376 2023-04-20 20:56:04,892 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 88c189b56fae4ae48b6bb2966c0bd880, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682024163997 2023-04-20 20:56:04,894 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 9bce135cf7af4a1eb0191ad6b827bcd6(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:04,895 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:04,895 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=13, startTime=1682024164849; duration=0sec 2023-04-20 20:56:04,895 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:04,901 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#501 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:04,902 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/ab716b61c1334855a6a1e85f67a50c70 2023-04-20 20:56:04,909 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/b8c4b3c7478a423481d365d6fce43aa9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b8c4b3c7478a423481d365d6fce43aa9 2023-04-20 20:56:04,916 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into b8c4b3c7478a423481d365d6fce43aa9(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:04,916 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:04,916 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=13, startTime=1682024164850; duration=0sec 2023-04-20 20:56:04,916 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:04,918 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d545782f08c54a799ccb52e4411a45bc 2023-04-20 20:56:04,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/70727f565cec4198b7dc82173b7b47fc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/70727f565cec4198b7dc82173b7b47fc 2023-04-20 20:56:04,925 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/70727f565cec4198b7dc82173b7b47fc, entries=150, sequenceid=605, filesize=11.9 K 2023-04-20 20:56:04,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/ab716b61c1334855a6a1e85f67a50c70 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/ab716b61c1334855a6a1e85f67a50c70 2023-04-20 20:56:04,929 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/ab716b61c1334855a6a1e85f67a50c70, entries=150, sequenceid=605, filesize=11.9 K 2023-04-20 20:56:04,930 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/d545782f08c54a799ccb52e4411a45bc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d545782f08c54a799ccb52e4411a45bc 2023-04-20 20:56:04,936 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d545782f08c54a799ccb52e4411a45bc, entries=150, sequenceid=605, filesize=11.9 K 2023-04-20 20:56:04,937 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 57ms, sequenceid=605, compaction requested=false 2023-04-20 20:56:04,937 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:04,937 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:04,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:04,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:04,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:04,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,939 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:04,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a2d028[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:04,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:04,942 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:04,943 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,944 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:04,944 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:04,944 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:04,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,946 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:04,946 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,946 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:04,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:04,950 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:04,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:04,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:04,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:04,950 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:04,952 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:04,952 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,952 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:04,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,953 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,954 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:04,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:04,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:04,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:04,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:04,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:04,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:04,962 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:04,962 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:05,062 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:05,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:05,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:05,066 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:05,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:05,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:05,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,073 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,074 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:05,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:05,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:05,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:05,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:05,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:05,075 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:05,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:05,076 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,076 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:05,076 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:05,076 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,076 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:05,076 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:05,077 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,077 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,078 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,078 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:05,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f52e07[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:05,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:05,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,082 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:05,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:05,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:05,083 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:05,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:05,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:05,086 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:05,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:05,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:05,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:05,086 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:05,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,090 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,091 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:05,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,094 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:05,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,094 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:05,195 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:05,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:05,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:05,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:05,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:05,198 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:05,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:05,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:05,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,200 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,200 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,200 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,200 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:05,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:05,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:05,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:05,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:05,201 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:05,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:05,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:05,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:05,202 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:05,202 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:05,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:05,214 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:05,218 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/35e94df2838d4e5180ef682d8c0e3ad9 2023-04-20 20:56:05,228 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/a6491e9846fc449c8bf282f561d514d2 2023-04-20 20:56:05,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682024165220, max timestamp=1682024165233Num uniques 859; 2023-04-20 20:56:05,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:05,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682024165220, max timestamp=1682024165233Num uniques 859; 2023-04-20 20:56:05,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:05,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024165220, max timestamp=1682024165235Num uniques 859; 2023-04-20 20:56:05,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:05,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682024165246, max timestamp=1682024165265Num uniques 1068; 2023-04-20 20:56:05,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-20 20:56:05,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682024165246, max timestamp=1682024165265Num uniques 1018; 2023-04-20 20:56:05,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-20 20:56:05,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682024165246, max timestamp=1682024165265Num uniques 1068; 2023-04-20 20:56:05,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-20 20:56:05,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024165281, max timestamp=1682024165288Num uniques 1227; 2023-04-20 20:56:05,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024165281, max timestamp=1682024165288Num uniques 1227; 2023-04-20 20:56:05,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-20 20:56:05,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-20 20:56:05,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024165281, max timestamp=1682024165289Num uniques 1227; 2023-04-20 20:56:05,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=10392, suffix heap size=155576,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=227 2023-04-20 20:56:05,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024165306, max timestamp=1682024165315Num uniques 1086; 2023-04-20 20:56:05,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024165306, max timestamp=1682024165315Num uniques 1086; 2023-04-20 20:56:05,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:56:05,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:56:05,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024165306, max timestamp=1682024165315Num uniques 1086; 2023-04-20 20:56:05,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:56:05,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1682024165335, max timestamp=1682024165343Num uniques 1095; 2023-04-20 20:56:05,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1682024165335, max timestamp=1682024165343Num uniques 1095; 2023-04-20 20:56:05,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=11210, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-20 20:56:05,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=11210, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-20 20:56:05,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1682024165335, max timestamp=1682024165343Num uniques 1095; 2023-04-20 20:56:05,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=11210, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-20 20:56:05,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682024165358, max timestamp=1682024165373Num uniques 1104; 2023-04-20 20:56:05,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682024165358, max timestamp=1682024165373Num uniques 1104; 2023-04-20 20:56:05,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-20 20:56:05,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-20 20:56:05,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682024165358, max timestamp=1682024165373Num uniques 1104; 2023-04-20 20:56:05,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-20 20:56:05,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682024165393, max timestamp=1682024165400Num uniques 1063; 2023-04-20 20:56:05,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-20 20:56:05,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682024165393, max timestamp=1682024165400Num uniques 1063; 2023-04-20 20:56:05,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-20 20:56:05,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682024165393, max timestamp=1682024165400Num uniques 1063; 2023-04-20 20:56:05,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-20 20:56:05,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024165419, max timestamp=1682024165424Num uniques 1172; 2023-04-20 20:56:05,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024165419, max timestamp=1682024165424Num uniques 1172; 2023-04-20 20:56:05,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024165419, max timestamp=1682024165424Num uniques 1172; 2023-04-20 20:56:05,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-20 20:56:05,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-20 20:56:05,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-20 20:56:05,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682024165439, max timestamp=1682024165451Num uniques 1131; 2023-04-20 20:56:05,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682024165439, max timestamp=1682024165451Num uniques 1131; 2023-04-20 20:56:05,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-20 20:56:05,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-20 20:56:05,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682024165439, max timestamp=1682024165451Num uniques 1131; 2023-04-20 20:56:05,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-20 20:56:05,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682024165474, max timestamp=1682024165485Num uniques 1140; 2023-04-20 20:56:05,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682024165474, max timestamp=1682024165485Num uniques 1140; 2023-04-20 20:56:05,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-20 20:56:05,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-20 20:56:05,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682024165474, max timestamp=1682024165485Num uniques 1140; 2023-04-20 20:56:05,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-20 20:56:05,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1682024165499, max timestamp=1682024165513Num uniques 1149; 2023-04-20 20:56:05,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1682024165499, max timestamp=1682024165513Num uniques 1149; 2023-04-20 20:56:05,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=18274, suffix heap size=147464,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=399 2023-04-20 20:56:05,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=18274, suffix heap size=147464,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=399 2023-04-20 20:56:05,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1682024165499, max timestamp=1682024165513Num uniques 1149; 2023-04-20 20:56:05,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=18274, suffix heap size=147464,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=399 2023-04-20 20:56:05,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=358, cellSize=16390, totalHeapSize=37472, min timestamp=1682024165528, max timestamp=1682024165542Num uniques 1258; 2023-04-20 20:56:05,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=358, cellSize=16390, totalHeapSize=37472, min timestamp=1682024165528, max timestamp=1682024165542Num uniques 1258; 2023-04-20 20:56:05,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=57610, new segment data size=16390, suffix heap size=158800,new segment heap size=37472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1258, new segment cells count=358 2023-04-20 20:56:05,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=57610, new segment data size=16390, suffix heap size=158800,new segment heap size=37472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1258, new segment cells count=358 2023-04-20 20:56:05,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682024165528, max timestamp=1682024165542Num uniques 1258; 2023-04-20 20:56:05,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=57610, new segment data size=7230, suffix heap size=158800,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1258, new segment cells count=158 2023-04-20 20:56:05,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024165552, max timestamp=1682024165566Num uniques 1017; 2023-04-20 20:56:05,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-20 20:56:05,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024165552, max timestamp=1682024165566Num uniques 1217; 2023-04-20 20:56:05,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=55732, new segment data size=9932, suffix heap size=154536,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1217, new segment cells count=217 2023-04-20 20:56:05,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024165552, max timestamp=1682024165566Num uniques 1217; 2023-04-20 20:56:05,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=55732, new segment data size=9932, suffix heap size=154536,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1217, new segment cells count=217 2023-04-20 20:56:05,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1682024165579, max timestamp=1682024165594Num uniques 1076; 2023-04-20 20:56:05,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-20 20:56:05,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1682024165579, max timestamp=1682024165594Num uniques 1076; 2023-04-20 20:56:05,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-20 20:56:05,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1682024165579, max timestamp=1682024165594Num uniques 1076; 2023-04-20 20:56:05,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-20 20:56:05,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1682024165609, max timestamp=1682024165618Num uniques 1135; 2023-04-20 20:56:05,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-04-20 20:56:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1682024165609, max timestamp=1682024165618Num uniques 1135; 2023-04-20 20:56:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-04-20 20:56:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1682024165609, max timestamp=1682024165618Num uniques 1135; 2023-04-20 20:56:05,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-04-20 20:56:05,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682024165618, max timestamp=1682024165641Num uniques 1194; 2023-04-20 20:56:05,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682024165618, max timestamp=1682024165641Num uniques 1194; 2023-04-20 20:56:05,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-20 20:56:05,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682024165618, max timestamp=1682024165641Num uniques 1194; 2023-04-20 20:56:05,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-20 20:56:05,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-20 20:56:05,644 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/2527df9b5a944ad990a6f753cb638d9d 2023-04-20 20:56:05,648 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/35e94df2838d4e5180ef682d8c0e3ad9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/35e94df2838d4e5180ef682d8c0e3ad9 2023-04-20 20:56:05,653 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/35e94df2838d4e5180ef682d8c0e3ad9, entries=50, sequenceid=613, filesize=7.1 K 2023-04-20 20:56:05,654 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/a6491e9846fc449c8bf282f561d514d2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a6491e9846fc449c8bf282f561d514d2 2023-04-20 20:56:05,657 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a6491e9846fc449c8bf282f561d514d2, entries=50, sequenceid=613, filesize=7.1 K 2023-04-20 20:56:05,657 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/2527df9b5a944ad990a6f753cb638d9d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2527df9b5a944ad990a6f753cb638d9d 2023-04-20 20:56:05,663 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2527df9b5a944ad990a6f753cb638d9d, entries=50, sequenceid=613, filesize=7.1 K 2023-04-20 20:56:05,664 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=93.93 KB/96180 for 8552cf5289bf62dd18e69f1f78b0b5fd in 462ms, sequenceid=613, compaction requested=true 2023-04-20 20:56:05,664 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:05,664 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:05,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:05,664 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=100.63 KB heapSize=277.41 KB 2023-04-20 20:56:05,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:05,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:05,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:05,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:05,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:05,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:05,666 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103738ea[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:05,668 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:05,668 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,670 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:05,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:05,670 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:05,670 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,670 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:05,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:05,671 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/8a6cba927ae34afb82c6aa0c44cd8a73 2023-04-20 20:56:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:05,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:05,677 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:05,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:05,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:05,677 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:05,677 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:05,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,683 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:05,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,685 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:05,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,685 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:05,685 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024225685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024225685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024225686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024225697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024225698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,786 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:05,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:05,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:05,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:05,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:05,789 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:05,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:05,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:05,789 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:05,793 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:05,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:05,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:05,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:05,795 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:05,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:05,797 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,797 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,797 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,797 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:05,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76379b07[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:05,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:05,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:05,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:05,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:05,802 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,802 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:05,803 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:05,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,803 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024225799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024225799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024225799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024225799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:05,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:05,806 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:05,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:05,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:05,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:05,806 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:05,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,809 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,812 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,812 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,812 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,812 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:05,812 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:05,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:05,812 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,812 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,813 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:05,813 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:05,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:05,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024225808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,913 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:05,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:05,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:05,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:05,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:05,916 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:05,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,916 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:05,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:05,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,917 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,920 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,920 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,921 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,921 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:05,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:05,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:05,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:05,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:05,922 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:05,922 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:05,923 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:05,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,925 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:05,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:05,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29cb2679[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:05,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:05,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,928 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:05,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:05,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:05,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,929 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:05,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:05,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:05,934 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:05,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:05,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:05,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:05,935 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:05,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:05,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,938 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:05,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:05,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,942 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:05,943 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:05,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:05,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:05,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:05,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:05,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:05,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:05,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:05,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:05,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:05,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:05,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:05,944 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:05,944 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024226006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024226006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024226006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024226007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024226020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,044 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:06,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:06,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:06,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:06,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:06,048 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:06,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:06,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:06,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,048 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,049 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:06,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:06,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:06,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:06,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:06,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:06,051 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:06,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,053 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:06,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799ea051[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:06,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:06,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,056 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:06,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:06,056 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:06,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:06,058 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:06,060 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:06,060 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:06,061 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:06,061 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:06,061 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:06,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,062 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,063 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,063 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,066 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:06,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:06,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:06,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,067 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:06,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,067 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/2a888c51d17545d89decf59dc99e0b32 2023-04-20 20:56:06,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=901 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/7b1bed549fa943839af431085061b6ba 2023-04-20 20:56:06,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/8a6cba927ae34afb82c6aa0c44cd8a73 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8a6cba927ae34afb82c6aa0c44cd8a73 2023-04-20 20:56:06,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8a6cba927ae34afb82c6aa0c44cd8a73, entries=300, sequenceid=901, filesize=19.0 K 2023-04-20 20:56:06,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/2a888c51d17545d89decf59dc99e0b32 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2a888c51d17545d89decf59dc99e0b32 2023-04-20 20:56:06,109 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2a888c51d17545d89decf59dc99e0b32, entries=150, sequenceid=901, filesize=11.9 K 2023-04-20 20:56:06,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/7b1bed549fa943839af431085061b6ba as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7b1bed549fa943839af431085061b6ba 2023-04-20 20:56:06,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7b1bed549fa943839af431085061b6ba, entries=150, sequenceid=901, filesize=11.9 K 2023-04-20 20:56:06,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~277.36 KB/284016, currentSize=80.51 KB/82440 for 8552cf5289bf62dd18e69f1f78b0b5fd in 449ms, sequenceid=901, compaction requested=true 2023-04-20 20:56:06,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:06,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:06,114 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:06,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:06,114 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:06,115 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:06,115 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:56:06,115 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52438 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:06,115 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,115 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:56:06,115 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,115 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9bce135cf7af4a1eb0191ad6b827bcd6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/ab716b61c1334855a6a1e85f67a50c70, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a6491e9846fc449c8bf282f561d514d2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2a888c51d17545d89decf59dc99e0b32] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=44.1 K 2023-04-20 20:56:06,115 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/915f511d5dcc49a68025cd1c54ac80d0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/70727f565cec4198b7dc82173b7b47fc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/35e94df2838d4e5180ef682d8c0e3ad9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8a6cba927ae34afb82c6aa0c44cd8a73] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=51.2 K 2023-04-20 20:56:06,117 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 915f511d5dcc49a68025cd1c54ac80d0, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682024163997 2023-04-20 20:56:06,117 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9bce135cf7af4a1eb0191ad6b827bcd6, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682024163997 2023-04-20 20:56:06,117 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 70727f565cec4198b7dc82173b7b47fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682024164067 2023-04-20 20:56:06,117 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ab716b61c1334855a6a1e85f67a50c70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682024164067 2023-04-20 20:56:06,118 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 35e94df2838d4e5180ef682d8c0e3ad9, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1682024165199 2023-04-20 20:56:06,118 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting a6491e9846fc449c8bf282f561d514d2, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1682024165199 2023-04-20 20:56:06,118 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8a6cba927ae34afb82c6aa0c44cd8a73, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682024165648 2023-04-20 20:56:06,118 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2a888c51d17545d89decf59dc99e0b32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682024165649 2023-04-20 20:56:06,124 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#509 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:06,125 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#510 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:06,140 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/eb87cce4f270421f97aa62363ab786ab as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/eb87cce4f270421f97aa62363ab786ab 2023-04-20 20:56:06,142 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/77831d6e882440e0a5714cae72faeb74 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/77831d6e882440e0a5714cae72faeb74 2023-04-20 20:56:06,145 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into eb87cce4f270421f97aa62363ab786ab(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:06,145 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,145 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=12, startTime=1682024166114; duration=0sec 2023-04-20 20:56:06,145 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:06,145 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:06,146 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 77831d6e882440e0a5714cae72faeb74(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:06,146 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,146 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=12, startTime=1682024166114; duration=0sec 2023-04-20 20:56:06,146 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:06,148 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:06,148 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:56:06,148 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,148 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b8c4b3c7478a423481d365d6fce43aa9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d545782f08c54a799ccb52e4411a45bc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2527df9b5a944ad990a6f753cb638d9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7b1bed549fa943839af431085061b6ba] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=44.1 K 2023-04-20 20:56:06,148 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b8c4b3c7478a423481d365d6fce43aa9, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682024163997 2023-04-20 20:56:06,149 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d545782f08c54a799ccb52e4411a45bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682024164067 2023-04-20 20:56:06,149 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2527df9b5a944ad990a6f753cb638d9d, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1682024165199 2023-04-20 20:56:06,149 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 7b1bed549fa943839af431085061b6ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682024165649 2023-04-20 20:56:06,155 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#511 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:06,167 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:06,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:06,170 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/95226192a8074be1837e9499b32be940 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/95226192a8074be1837e9499b32be940 2023-04-20 20:56:06,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:06,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:06,173 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:06,173 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,173 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:06,173 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:06,173 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,173 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,175 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into 95226192a8074be1837e9499b32be940(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:06,175 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,175 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=12, startTime=1682024166114; duration=0sec 2023-04-20 20:56:06,175 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,175 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:06,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:06,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:06,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:06,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:06,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:06,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:06,177 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:06,177 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,177 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:06,177 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 20:56:06,177 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:06,177 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:06,177 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:06,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:06,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:06,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:06,198 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/8931e39300254f91acda58dcb2304814 2023-04-20 20:56:06,210 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/12ae8ee1d2554ebb963ac0ef8921d087 2023-04-20 20:56:06,220 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/b390baf9060b45f29c31d8e0a5581578 2023-04-20 20:56:06,225 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/8931e39300254f91acda58dcb2304814 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8931e39300254f91acda58dcb2304814 2023-04-20 20:56:06,229 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8931e39300254f91acda58dcb2304814, entries=150, sequenceid=919, filesize=11.9 K 2023-04-20 20:56:06,229 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/12ae8ee1d2554ebb963ac0ef8921d087 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/12ae8ee1d2554ebb963ac0ef8921d087 2023-04-20 20:56:06,239 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/12ae8ee1d2554ebb963ac0ef8921d087, entries=150, sequenceid=919, filesize=11.9 K 2023-04-20 20:56:06,239 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/b390baf9060b45f29c31d8e0a5581578 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b390baf9060b45f29c31d8e0a5581578 2023-04-20 20:56:06,242 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b390baf9060b45f29c31d8e0a5581578, entries=150, sequenceid=919, filesize=11.9 K 2023-04-20 20:56:06,243 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 66ms, sequenceid=919, compaction requested=false 2023-04-20 20:56:06,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,243 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:06,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:06,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:06,243 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:06,255 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,256 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,256 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:06,256 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c737e0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:06,258 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:06,258 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,259 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:06,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:06,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:06,261 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:06,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:06,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:06,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:06,262 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:06,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:06,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:06,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:06,262 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:06,263 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,263 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,263 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,264 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,267 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:06,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,271 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:06,271 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:06,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:06,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:06,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:06,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:06,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:06,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:06,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:06,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=930 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/6017f72182114dbb9e29f097597a14df 2023-04-20 20:56:06,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=930 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/4479d4c676bf4a75a68755f715a1d2a5 2023-04-20 20:56:06,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024166337, max timestamp=1682024166355Num uniques 859; 2023-04-20 20:56:06,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:06,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024166337, max timestamp=1682024166355Num uniques 859; 2023-04-20 20:56:06,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:06,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024166337, max timestamp=1682024166355Num uniques 859; 2023-04-20 20:56:06,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:06,371 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:06,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:06,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:06,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:06,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:06,374 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:06,375 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,375 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:06,375 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:06,375 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,375 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,376 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,377 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:06,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:06,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:06,379 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:06,379 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,379 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:06,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:06,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:06,380 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,380 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:06,380 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:06,380 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:06,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:06,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:06,380 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:06,382 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,382 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ae2615c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:06,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:06,384 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,388 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:06,389 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,390 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:06,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:06,390 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,390 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:06,393 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:06,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:06,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:06,393 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:06,393 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:06,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024226380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024226381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024226382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024226382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024226382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,395 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,396 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,398 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:06,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:06,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,401 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:06,401 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,401 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,402 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,402 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,402 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,402 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024226495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024226495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024226495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024226495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024226496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,501 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:06,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:06,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:06,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:06,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:06,504 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:06,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:06,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:06,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,505 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,508 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,508 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,508 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,508 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:06,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:06,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:06,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:06,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:06,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:06,509 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:06,509 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,509 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:06,509 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:06,510 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:06,511 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,511 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,511 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,511 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aea7497[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:06,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:06,513 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,514 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:06,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:06,514 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:06,514 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,514 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:06,516 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:06,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:06,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:06,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:06,519 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:06,520 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:06,520 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:06,520 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:06,520 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:06,522 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,522 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,522 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,522 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,524 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:06,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,527 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:06,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:06,527 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:06,527 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,527 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,528 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,627 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:06,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:06,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:06,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:06,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:06,631 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:06,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:06,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:06,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,631 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,633 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,633 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,633 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,633 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:06,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:06,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:06,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:06,634 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:06,634 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,634 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:06,635 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:06,635 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:06,635 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,635 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:06,635 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:06,635 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,635 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:06,635 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:06,635 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,635 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:06,635 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:06,637 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,637 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,637 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,637 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66faddb3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:06,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:06,640 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,642 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,643 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:06,644 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:06,644 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,644 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,644 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:06,647 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:06,647 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:06,647 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:06,647 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:06,647 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:06,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,649 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,650 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,651 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,653 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,653 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,653 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,653 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,653 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:06,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:06,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:06,653 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,654 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,654 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,654 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,655 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:06,655 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024226700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024226701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024226701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024226702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:06,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024226702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=930 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/e10717d2c6774736a239c5e343254429 2023-04-20 20:56:06,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/6017f72182114dbb9e29f097597a14df as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/6017f72182114dbb9e29f097597a14df 2023-04-20 20:56:06,755 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:06,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:06,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:06,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:06,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:06,759 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:06,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/6017f72182114dbb9e29f097597a14df, entries=250, sequenceid=930, filesize=16.7 K 2023-04-20 20:56:06,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:06,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:06,759 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,760 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/4479d4c676bf4a75a68755f715a1d2a5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4479d4c676bf4a75a68755f715a1d2a5 2023-04-20 20:56:06,762 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,762 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,762 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,762 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:06,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4479d4c676bf4a75a68755f715a1d2a5, entries=150, sequenceid=930, filesize=11.9 K 2023-04-20 20:56:06,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:06,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/e10717d2c6774736a239c5e343254429 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e10717d2c6774736a239c5e343254429 2023-04-20 20:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:06,765 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,765 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:06,765 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:06,765 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:06,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:06,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:06,765 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:06,768 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,768 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,769 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:06,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:06,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6b9ebb[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:06,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:06,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e10717d2c6774736a239c5e343254429, entries=150, sequenceid=930, filesize=11.9 K 2023-04-20 20:56:06,771 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:06,771 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:06,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8552cf5289bf62dd18e69f1f78b0b5fd in 451ms, sequenceid=930, compaction requested=true 2023-04-20 20:56:06,772 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:06,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,772 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,772 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:06,772 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:06,772 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:06,772 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:06,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:06,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:06,773 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42834 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:06,773 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:56:06,773 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,773 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:06,773 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/eb87cce4f270421f97aa62363ab786ab, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8931e39300254f91acda58dcb2304814, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/6017f72182114dbb9e29f097597a14df] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=41.8 K 2023-04-20 20:56:06,773 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:56:06,773 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,774 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/77831d6e882440e0a5714cae72faeb74, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/12ae8ee1d2554ebb963ac0ef8921d087, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4479d4c676bf4a75a68755f715a1d2a5] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=37.1 K 2023-04-20 20:56:06,775 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 77831d6e882440e0a5714cae72faeb74, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682024165649 2023-04-20 20:56:06,775 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting eb87cce4f270421f97aa62363ab786ab, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682024165649 2023-04-20 20:56:06,775 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 12ae8ee1d2554ebb963ac0ef8921d087, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1682024165683 2023-04-20 20:56:06,775 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8931e39300254f91acda58dcb2304814, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1682024165683 2023-04-20 20:56:06,775 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4479d4c676bf4a75a68755f715a1d2a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=930, earliestPutTs=1682024166313 2023-04-20 20:56:06,776 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6017f72182114dbb9e29f097597a14df, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=930, earliestPutTs=1682024166313 2023-04-20 20:56:06,776 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:06,781 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:06,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:06,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:06,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:06,781 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:06,783 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,783 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,784 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,784 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,785 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,787 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:06,787 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#519 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,796 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:06,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,797 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:06,797 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:06,804 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/2bbae44c59794d4b8f2ba6b1790710d8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2bbae44c59794d4b8f2ba6b1790710d8 2023-04-20 20:56:06,810 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into 2bbae44c59794d4b8f2ba6b1790710d8(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:06,811 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,811 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=13, startTime=1682024166772; duration=0sec 2023-04-20 20:56:06,811 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:06,811 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:06,812 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:06,813 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:56:06,813 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,813 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/95226192a8074be1837e9499b32be940, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b390baf9060b45f29c31d8e0a5581578, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e10717d2c6774736a239c5e343254429] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=37.1 K 2023-04-20 20:56:06,813 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 95226192a8074be1837e9499b32be940, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=901, earliestPutTs=1682024165649 2023-04-20 20:56:06,813 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b390baf9060b45f29c31d8e0a5581578, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1682024165683 2023-04-20 20:56:06,814 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e10717d2c6774736a239c5e343254429, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=930, earliestPutTs=1682024166313 2023-04-20 20:56:06,819 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#520 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:06,827 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/20f7a85189064e8e894929d77d2ae2fb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/20f7a85189064e8e894929d77d2ae2fb 2023-04-20 20:56:06,831 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into 20f7a85189064e8e894929d77d2ae2fb(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:06,831 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,831 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=13, startTime=1682024166773; duration=0sec 2023-04-20 20:56:06,831 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:06,897 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:06,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:06,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:06,900 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:06,900 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:06,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,902 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:06,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:06,904 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:06,912 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/49cf7bbca7a947d08b32a6cc2f53a473 2023-04-20 20:56:06,924 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/382995d49fc64808a65bd47289031531 2023-04-20 20:56:06,933 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/348e926513c5433fbacb77ae90acddb8 2023-04-20 20:56:06,938 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/49cf7bbca7a947d08b32a6cc2f53a473 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/49cf7bbca7a947d08b32a6cc2f53a473 2023-04-20 20:56:06,942 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/49cf7bbca7a947d08b32a6cc2f53a473, entries=150, sequenceid=967, filesize=11.9 K 2023-04-20 20:56:06,943 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/382995d49fc64808a65bd47289031531 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/382995d49fc64808a65bd47289031531 2023-04-20 20:56:06,947 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/382995d49fc64808a65bd47289031531, entries=150, sequenceid=967, filesize=11.9 K 2023-04-20 20:56:06,947 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/348e926513c5433fbacb77ae90acddb8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/348e926513c5433fbacb77ae90acddb8 2023-04-20 20:56:06,951 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/348e926513c5433fbacb77ae90acddb8, entries=150, sequenceid=967, filesize=11.9 K 2023-04-20 20:56:06,952 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 48ms, sequenceid=967, compaction requested=false 2023-04-20 20:56:06,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:06,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:06,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:06,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:06,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:06,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:06,955 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,956 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68ae68b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:06,958 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:06,958 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,960 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,960 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,960 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:06,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:06,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:06,960 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:06,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:06,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,963 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:06,963 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:06,965 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:06,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:06,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:06,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:06,966 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:06,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:06,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:06,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:06,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:06,971 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:06,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:06,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:06,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:06,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:06,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:06,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:06,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:06,974 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:06,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:06,975 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:06,975 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:06,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:07,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:07,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/d9ba039616a84650a4319c0f4847ee17 2023-04-20 20:56:07,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/c43e0fa13a414572a472af2ab6fee0d4 2023-04-20 20:56:07,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024227049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024227057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024167046, max timestamp=1682024167057Num uniques 859; 2023-04-20 20:56:07,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:07,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024167046, max timestamp=1682024167057Num uniques 859; 2023-04-20 20:56:07,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:07,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024167046, max timestamp=1682024167057Num uniques 859; 2023-04-20 20:56:07,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:07,075 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:07,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:07,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:07,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:07,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:07,078 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:07,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:07,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:07,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,078 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,079 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,080 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:07,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:07,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:07,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:07,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:07,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:07,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:07,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,083 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:07,083 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:07,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:07,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7d7401[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:07,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:07,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,087 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:07,087 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:07,087 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:07,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:07,092 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:07,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:07,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:07,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:07,092 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:07,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,097 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:07,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:07,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:07,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,098 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:07,098 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:07,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024227093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024227098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024227099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024227161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024227161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,198 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:07,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:07,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:07,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:07,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:07,201 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:07,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:07,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:07,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,203 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,203 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:07,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:07,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:07,205 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,205 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:07,205 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:07,205 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:07,207 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,207 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:07,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fda2118[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:07,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:07,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:07,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024227206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,211 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:07,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:07,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024227206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024227207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:07,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,212 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:07,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,215 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/066c90d067f64b9aaa5e29c1b89a5dcf as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/066c90d067f64b9aaa5e29c1b89a5dcf 2023-04-20 20:56:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:07,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:07,215 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:07,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:07,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:07,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:07,215 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:07,217 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,217 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,217 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,217 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,219 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,219 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into 066c90d067f64b9aaa5e29c1b89a5dcf(size=13.4 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:07,219 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:07,219 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=13, startTime=1682024166772; duration=0sec 2023-04-20 20:56:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,219 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,225 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,225 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,226 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:07,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:07,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:07,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,226 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:07,226 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:07,326 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:07,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:07,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:07,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:07,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:07,329 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:07,329 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,329 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:07,329 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:07,330 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,330 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,331 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,331 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,331 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:07,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:07,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:07,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:07,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:07,333 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c97e81e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:07,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:07,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,338 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,338 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:07,338 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:07,338 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:07,338 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,338 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:07,341 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:07,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,341 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,341 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:07,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:07,346 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:07,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:07,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:07,346 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:07,346 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:07,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,349 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,350 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,352 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:07,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:07,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,353 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:07,353 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:07,353 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,353 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,353 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,353 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:07,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024227370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024227370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024227412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024227413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024227414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,454 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:07,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:07,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:07,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:07,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:07,458 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:07,458 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,458 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:07,458 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:07,458 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,458 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,460 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,460 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,460 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,460 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:07,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:07,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:07,461 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:07,462 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,462 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:07,462 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:07,462 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,463 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:07,463 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:07,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,465 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,466 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,466 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/41188177cc9645c186aa89f4de7d9aaa 2023-04-20 20:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@df8e3b9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:07,468 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:07,468 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,471 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:07,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:07,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:07,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:07,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/d9ba039616a84650a4319c0f4847ee17 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/d9ba039616a84650a4319c0f4847ee17 2023-04-20 20:56:07,472 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:07,472 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,472 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,472 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:07,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:07,475 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:07,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:07,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:07,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:07,475 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:07,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/d9ba039616a84650a4319c0f4847ee17, entries=250, sequenceid=978, filesize=16.7 K 2023-04-20 20:56:07,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/c43e0fa13a414572a472af2ab6fee0d4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c43e0fa13a414572a472af2ab6fee0d4 2023-04-20 20:56:07,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:07,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:07,477 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:07,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:07,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:07,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:07,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c43e0fa13a414572a472af2ab6fee0d4, entries=150, sequenceid=978, filesize=11.9 K 2023-04-20 20:56:07,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/41188177cc9645c186aa89f4de7d9aaa as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/41188177cc9645c186aa89f4de7d9aaa 2023-04-20 20:56:07,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:07,481 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:07,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:07,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:07,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:07,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,482 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:07,482 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:07,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:07,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:07,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/41188177cc9645c186aa89f4de7d9aaa, entries=150, sequenceid=978, filesize=11.9 K 2023-04-20 20:56:07,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8552cf5289bf62dd18e69f1f78b0b5fd in 471ms, sequenceid=978, compaction requested=true 2023-04-20 20:56:07,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:07,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:07,485 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:07,485 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:07,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:07,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:07,486 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42936 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:07,486 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:07,486 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/A is initiating minor compaction (all files) 2023-04-20 20:56:07,486 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/B is initiating minor compaction (all files) 2023-04-20 20:56:07,486 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/A in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,486 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/B in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,486 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2bbae44c59794d4b8f2ba6b1790710d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/49cf7bbca7a947d08b32a6cc2f53a473, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/d9ba039616a84650a4319c0f4847ee17] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=41.9 K 2023-04-20 20:56:07,486 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/066c90d067f64b9aaa5e29c1b89a5dcf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/382995d49fc64808a65bd47289031531, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c43e0fa13a414572a472af2ab6fee0d4] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=37.2 K 2023-04-20 20:56:07,487 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2bbae44c59794d4b8f2ba6b1790710d8, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=930, earliestPutTs=1682024166313 2023-04-20 20:56:07,487 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 066c90d067f64b9aaa5e29c1b89a5dcf, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=930, earliestPutTs=1682024166313 2023-04-20 20:56:07,487 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 49cf7bbca7a947d08b32a6cc2f53a473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1682024166380 2023-04-20 20:56:07,487 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 382995d49fc64808a65bd47289031531, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1682024166380 2023-04-20 20:56:07,487 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d9ba039616a84650a4319c0f4847ee17, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1682024167008 2023-04-20 20:56:07,487 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c43e0fa13a414572a472af2ab6fee0d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1682024167011 2023-04-20 20:56:07,494 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#B#compaction#528 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:07,494 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#A#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:07,510 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/e767ad35eea94b19810b218bf62281f8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/e767ad35eea94b19810b218bf62281f8 2023-04-20 20:56:07,514 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/A of 8552cf5289bf62dd18e69f1f78b0b5fd into e767ad35eea94b19810b218bf62281f8(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:07,514 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:07,514 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/A, priority=13, startTime=1682024167485; duration=0sec 2023-04-20 20:56:07,514 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:07,514 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:07,516 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:07,517 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 8552cf5289bf62dd18e69f1f78b0b5fd/C is initiating minor compaction (all files) 2023-04-20 20:56:07,517 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8552cf5289bf62dd18e69f1f78b0b5fd/C in TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,517 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/20f7a85189064e8e894929d77d2ae2fb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/348e926513c5433fbacb77ae90acddb8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/41188177cc9645c186aa89f4de7d9aaa] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp, totalSize=37.2 K 2023-04-20 20:56:07,517 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 20f7a85189064e8e894929d77d2ae2fb, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=930, earliestPutTs=1682024166313 2023-04-20 20:56:07,517 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 348e926513c5433fbacb77ae90acddb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1682024166380 2023-04-20 20:56:07,518 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 41188177cc9645c186aa89f4de7d9aaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1682024167011 2023-04-20 20:56:07,522 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8552cf5289bf62dd18e69f1f78b0b5fd#C#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:07,582 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:07,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:07,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:07,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:07,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:07,587 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:07,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:07,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:07,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:07,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:07,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,589 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:07,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:07,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:07,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:07,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:07,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:07,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:07,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:07,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:07,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:07,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:07,590 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:07,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:07,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. started... 2023-04-20 20:56:07,590 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 20:56:07,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:07,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:07,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:07,590 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:07,591 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:07,591 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:07,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:07,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. as already flushing 2023-04-20 20:56:07,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024227697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024227702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024227719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024227719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024227720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024227806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024227813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:07,906 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/af40726cf7be43609d6a995924a5a27f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/af40726cf7be43609d6a995924a5a27f 2023-04-20 20:56:07,910 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/B of 8552cf5289bf62dd18e69f1f78b0b5fd into af40726cf7be43609d6a995924a5a27f(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:07,910 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:07,910 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/B, priority=13, startTime=1682024167485; duration=0sec 2023-04-20 20:56:07,910 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:07,930 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/be982a1f0d30486dacf1f64745152161 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/be982a1f0d30486dacf1f64745152161 2023-04-20 20:56:07,934 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8552cf5289bf62dd18e69f1f78b0b5fd/C of 8552cf5289bf62dd18e69f1f78b0b5fd into be982a1f0d30486dacf1f64745152161(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:07,934 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:07,934 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd., storeName=8552cf5289bf62dd18e69f1f78b0b5fd/C, priority=13, startTime=1682024167485; duration=0sec 2023-04-20 20:56:07,934 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:08,003 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/ff40de839c204b31a488e70c277222a4 2023-04-20 20:56:08,014 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/9168ffdcfa1a46e5a0c1d6a65cf8b9ff 2023-04-20 20:56:08,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:08,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024228015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:08,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024228018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:08,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40066 deadline: 1682024228227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:08,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39986 deadline: 1682024228229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:08,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40004 deadline: 1682024228230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:08,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1682024228322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:08,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39984 deadline: 1682024228326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,426 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/879b53d5ab024a0fb303b756b7bdb5de 2023-04-20 20:56:08,430 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/ff40de839c204b31a488e70c277222a4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ff40de839c204b31a488e70c277222a4 2023-04-20 20:56:08,434 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ff40de839c204b31a488e70c277222a4, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-20 20:56:08,435 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/9168ffdcfa1a46e5a0c1d6a65cf8b9ff as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9168ffdcfa1a46e5a0c1d6a65cf8b9ff 2023-04-20 20:56:08,440 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9168ffdcfa1a46e5a0c1d6a65cf8b9ff, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-20 20:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/879b53d5ab024a0fb303b756b7bdb5de as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/879b53d5ab024a0fb303b756b7bdb5de 2023-04-20 20:56:08,443 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/879b53d5ab024a0fb303b756b7bdb5de, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-20 20:56:08,444 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 8552cf5289bf62dd18e69f1f78b0b5fd in 854ms, sequenceid=1015, compaction requested=false 2023-04-20 20:56:08,444 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:08,444 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:08,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:08,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:08,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:08,444 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:08,446 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:08,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f1537b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:08,448 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:08,448 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,449 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:08,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:08,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:08,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,452 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:08,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:08,454 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:08,454 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:08,454 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:08,456 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:08,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:08,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:08,457 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:08,457 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:08,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:08,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:08,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:08,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:08,460 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:08,461 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:08,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:08,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:08,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:08,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:08,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:08,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:08,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:08,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:08,465 DEBUG [Thread-3018] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:08,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:08,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:08,465 DEBUG [Thread-3018] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:08,466 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,466 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:08,466 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:08,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:08,564 DEBUG [Thread-3023] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:62055 2023-04-20 20:56:08,564 DEBUG [Thread-3023] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:08,564 DEBUG [Thread-3027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ac98310 to 127.0.0.1:62055 2023-04-20 20:56:08,565 DEBUG [Thread-3027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:08,566 DEBUG [Thread-3018] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:08,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:08,568 DEBUG [Thread-3019] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:62055 2023-04-20 20:56:08,568 DEBUG [Thread-3019] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:08,568 DEBUG [Thread-3025] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:62055 2023-04-20 20:56:08,568 DEBUG [Thread-3025] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:08,569 DEBUG [Thread-3021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:62055 2023-04-20 20:56:08,569 DEBUG [Thread-3021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:08,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:08,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:56:08,834 DEBUG [Thread-3016] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:62055 2023-04-20 20:56:08,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:08,834 DEBUG [Thread-3016] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:08,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:08,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:08,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:08,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:08,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:08,835 DEBUG [Thread-3010] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:62055 2023-04-20 20:56:08,835 DEBUG [Thread-3010] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:09,234 DEBUG [Thread-3012] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:62055 2023-04-20 20:56:09,234 DEBUG [Thread-3012] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:09,237 DEBUG [Thread-3014] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:62055 2023-04-20 20:56:09,237 DEBUG [Thread-3014] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:09,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/4af59abec87b454096fe95d98f2900c0 2023-04-20 20:56:09,241 DEBUG [Thread-3008] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:62055 2023-04-20 20:56:09,241 DEBUG [Thread-3008] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 183 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 175 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 184 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 171 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1839 2023-04-20 20:56:09,241 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 5517 rows 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1826 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 5478 rows 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1831 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 5489 rows 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1813 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 5439 rows 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(402): scanned 1825 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(403): verified 5475 rows 2023-04-20 20:56:09,242 INFO [Listener at localhost/45619] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 20:56:09,242 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:62055 2023-04-20 20:56:09,242 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:09,243 INFO [Listener at localhost/45619] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 20:56:09,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-20 20:56:09,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:09,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 20:56:09,248 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024169248"}]},"ts":"1682024169248"} 2023-04-20 20:56:09,249 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 20:56:09,251 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 20:56:09,252 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8552cf5289bf62dd18e69f1f78b0b5fd, UNASSIGN}] 2023-04-20 20:56:09,260 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8552cf5289bf62dd18e69f1f78b0b5fd, UNASSIGN 2023-04-20 20:56:09,260 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=8552cf5289bf62dd18e69f1f78b0b5fd, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:09,261 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024169260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024169260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024169260"}]},"ts":"1682024169260"} 2023-04-20 20:56:09,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:56:09,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 20:56:09,413 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:09,413 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8552cf5289bf62dd18e69f1f78b0b5fd, disabling compactions & flushes 2023-04-20 20:56:09,413 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:09,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 20:56:09,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/c1353808cf684f3e9b384dc29352fe8f 2023-04-20 20:56:09,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 20:56:10,071 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/1fa665a1eeb644ebbb784655361e26d7 2023-04-20 20:56:10,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/4af59abec87b454096fe95d98f2900c0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4af59abec87b454096fe95d98f2900c0 2023-04-20 20:56:10,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4af59abec87b454096fe95d98f2900c0, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-20 20:56:10,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/c1353808cf684f3e9b384dc29352fe8f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c1353808cf684f3e9b384dc29352fe8f 2023-04-20 20:56:10,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c1353808cf684f3e9b384dc29352fe8f, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-20 20:56:10,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/1fa665a1eeb644ebbb784655361e26d7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/1fa665a1eeb644ebbb784655361e26d7 2023-04-20 20:56:10,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/1fa665a1eeb644ebbb784655361e26d7, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-20 20:56:10,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 8552cf5289bf62dd18e69f1f78b0b5fd in 1251ms, sequenceid=1029, compaction requested=true 2023-04-20 20:56:10,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:10,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:10,085 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. because compaction request was cancelled 2023-04-20 20:56:10,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:10,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:10,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. after waiting 0 ms 2023-04-20 20:56:10,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:10,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:10,085 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. because compaction request was cancelled 2023-04-20 20:56:10,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8552cf5289bf62dd18e69f1f78b0b5fd 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 20:56:10,085 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. because compaction request was cancelled 2023-04-20 20:56:10,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:10,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=A 2023-04-20 20:56:10,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:10,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=B 2023-04-20 20:56:10,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:10,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8552cf5289bf62dd18e69f1f78b0b5fd, store=C 2023-04-20 20:56:10,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:10,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 20:56:10,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/250022f28ed3428d8e04141e61403d9e 2023-04-20 20:56:10,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/86e84da0fb1b4b77baa10b80a4774527 2023-04-20 20:56:11,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/3e7c328536554ffea532a611aa4037be 2023-04-20 20:56:11,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/A/250022f28ed3428d8e04141e61403d9e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/250022f28ed3428d8e04141e61403d9e 2023-04-20 20:56:11,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/250022f28ed3428d8e04141e61403d9e, entries=100, sequenceid=1036, filesize=9.5 K 2023-04-20 20:56:11,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/B/86e84da0fb1b4b77baa10b80a4774527 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86e84da0fb1b4b77baa10b80a4774527 2023-04-20 20:56:11,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86e84da0fb1b4b77baa10b80a4774527, entries=100, sequenceid=1036, filesize=9.5 K 2023-04-20 20:56:11,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/.tmp/C/3e7c328536554ffea532a611aa4037be as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3e7c328536554ffea532a611aa4037be 2023-04-20 20:56:11,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3e7c328536554ffea532a611aa4037be, entries=100, sequenceid=1036, filesize=9.5 K 2023-04-20 20:56:11,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 8552cf5289bf62dd18e69f1f78b0b5fd in 1239ms, sequenceid=1036, compaction requested=true 2023-04-20 20:56:11,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4d25dcef4cc34626956fad317658ddc0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/96a5c966f6924e34ba673019e1436082, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a5ebe368ddf94ab99470bcd93bfa5f9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/73551f1efabc4bc297a0d92cc3a6ca13, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/7833df0ef9bc4c5b887d911c9c047f30, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/9ea0f8dc98bd47bb8708f7d7d02b77d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/121669bb5bd44d1b8db17b1584c229c0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/296fb75a6a6b4ad4bd00717f187f1fa4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/423f0779615a4e5999d419efe1326309, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8ed404851810491da56fb2e5292c0b35, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/206162eac6014ac7bd240709a9f065af, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/070c95846a524ec4a0d2a81f6b55493d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2b75c60b6a8a45088d76a960ac0dc3c0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5c610fbf8b5d496b98a9280213171505, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/3444b46f57e74eeabb23bc1be339d2d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5d98ecda38e84b869354d3df2fec3817, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/bfce66a1a2de46d29ccdf4ddbea70c43, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b976904139e348f989af70350deccec8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/16c165d856c84451b60a18b686cf7328, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/86c82e6052804af58c04adec4511cfc1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/39554c9e31394031bd312bb466a08cc3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/736e0fb3e9e14383a36ac00343f51c01, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b0da2887c58043bca19ccb92508dd0d6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8136437e35d2496cb57aa7fc416dcc16, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/0e9fe819e1b8409ebf92c0be138389ce, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b1057428b2ae4a6a8a1e87071015a8ce, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/c78ad043fe1d46ecbda80108f01ea59c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a7d0d3d294a14a93b01b4dc51589330e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/87950f5b8a9c4c74b68dbbb62b5dae10, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b4d696a4062c449eb8028bff4589c67f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/fc90e4d0559442c2a481eb443b966b19, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5e4e803bba114fb19d000aa882e269db, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/01d379cc26884e00a621080bf5bcadf6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ed8ab8259ca0462aa682ea22ed5be2b0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/09fddda0bb9c4d6e84990ae57c3ad80d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/08939d4fd9aa49db9086bf21f65091d1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ee415c946ab145299215622f3d3f3926, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/915f511d5dcc49a68025cd1c54ac80d0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/70727f565cec4198b7dc82173b7b47fc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/35e94df2838d4e5180ef682d8c0e3ad9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8a6cba927ae34afb82c6aa0c44cd8a73, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/eb87cce4f270421f97aa62363ab786ab, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8931e39300254f91acda58dcb2304814, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/6017f72182114dbb9e29f097597a14df, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2bbae44c59794d4b8f2ba6b1790710d8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/49cf7bbca7a947d08b32a6cc2f53a473, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/d9ba039616a84650a4319c0f4847ee17] to archive 2023-04-20 20:56:11,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:56:11,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4d25dcef4cc34626956fad317658ddc0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4d25dcef4cc34626956fad317658ddc0 2023-04-20 20:56:11,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/96a5c966f6924e34ba673019e1436082 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/96a5c966f6924e34ba673019e1436082 2023-04-20 20:56:11,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a5ebe368ddf94ab99470bcd93bfa5f9d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a5ebe368ddf94ab99470bcd93bfa5f9d 2023-04-20 20:56:11,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/73551f1efabc4bc297a0d92cc3a6ca13 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/73551f1efabc4bc297a0d92cc3a6ca13 2023-04-20 20:56:11,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/7833df0ef9bc4c5b887d911c9c047f30 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/7833df0ef9bc4c5b887d911c9c047f30 2023-04-20 20:56:11,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/9ea0f8dc98bd47bb8708f7d7d02b77d8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/9ea0f8dc98bd47bb8708f7d7d02b77d8 2023-04-20 20:56:11,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/121669bb5bd44d1b8db17b1584c229c0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/121669bb5bd44d1b8db17b1584c229c0 2023-04-20 20:56:11,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/296fb75a6a6b4ad4bd00717f187f1fa4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/296fb75a6a6b4ad4bd00717f187f1fa4 2023-04-20 20:56:11,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/423f0779615a4e5999d419efe1326309 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/423f0779615a4e5999d419efe1326309 2023-04-20 20:56:11,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8ed404851810491da56fb2e5292c0b35 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8ed404851810491da56fb2e5292c0b35 2023-04-20 20:56:11,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/206162eac6014ac7bd240709a9f065af to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/206162eac6014ac7bd240709a9f065af 2023-04-20 20:56:11,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/070c95846a524ec4a0d2a81f6b55493d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/070c95846a524ec4a0d2a81f6b55493d 2023-04-20 20:56:11,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2b75c60b6a8a45088d76a960ac0dc3c0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2b75c60b6a8a45088d76a960ac0dc3c0 2023-04-20 20:56:11,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5c610fbf8b5d496b98a9280213171505 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5c610fbf8b5d496b98a9280213171505 2023-04-20 20:56:11,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/3444b46f57e74eeabb23bc1be339d2d8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/3444b46f57e74eeabb23bc1be339d2d8 2023-04-20 20:56:11,342 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5d98ecda38e84b869354d3df2fec3817 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5d98ecda38e84b869354d3df2fec3817 2023-04-20 20:56:11,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/bfce66a1a2de46d29ccdf4ddbea70c43 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/bfce66a1a2de46d29ccdf4ddbea70c43 2023-04-20 20:56:11,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b976904139e348f989af70350deccec8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b976904139e348f989af70350deccec8 2023-04-20 20:56:11,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/16c165d856c84451b60a18b686cf7328 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/16c165d856c84451b60a18b686cf7328 2023-04-20 20:56:11,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/86c82e6052804af58c04adec4511cfc1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/86c82e6052804af58c04adec4511cfc1 2023-04-20 20:56:11,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/39554c9e31394031bd312bb466a08cc3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/39554c9e31394031bd312bb466a08cc3 2023-04-20 20:56:11,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/736e0fb3e9e14383a36ac00343f51c01 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/736e0fb3e9e14383a36ac00343f51c01 2023-04-20 20:56:11,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b0da2887c58043bca19ccb92508dd0d6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b0da2887c58043bca19ccb92508dd0d6 2023-04-20 20:56:11,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8136437e35d2496cb57aa7fc416dcc16 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8136437e35d2496cb57aa7fc416dcc16 2023-04-20 20:56:11,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/0e9fe819e1b8409ebf92c0be138389ce to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/0e9fe819e1b8409ebf92c0be138389ce 2023-04-20 20:56:11,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b1057428b2ae4a6a8a1e87071015a8ce to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b1057428b2ae4a6a8a1e87071015a8ce 2023-04-20 20:56:11,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 20:56:11,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/c78ad043fe1d46ecbda80108f01ea59c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/c78ad043fe1d46ecbda80108f01ea59c 2023-04-20 20:56:11,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a7d0d3d294a14a93b01b4dc51589330e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/a7d0d3d294a14a93b01b4dc51589330e 2023-04-20 20:56:11,355 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/87950f5b8a9c4c74b68dbbb62b5dae10 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/87950f5b8a9c4c74b68dbbb62b5dae10 2023-04-20 20:56:11,356 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b4d696a4062c449eb8028bff4589c67f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/b4d696a4062c449eb8028bff4589c67f 2023-04-20 20:56:11,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/fc90e4d0559442c2a481eb443b966b19 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/fc90e4d0559442c2a481eb443b966b19 2023-04-20 20:56:11,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5e4e803bba114fb19d000aa882e269db to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/5e4e803bba114fb19d000aa882e269db 2023-04-20 20:56:11,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/01d379cc26884e00a621080bf5bcadf6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/01d379cc26884e00a621080bf5bcadf6 2023-04-20 20:56:11,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ed8ab8259ca0462aa682ea22ed5be2b0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ed8ab8259ca0462aa682ea22ed5be2b0 2023-04-20 20:56:11,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/09fddda0bb9c4d6e84990ae57c3ad80d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/09fddda0bb9c4d6e84990ae57c3ad80d 2023-04-20 20:56:11,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/08939d4fd9aa49db9086bf21f65091d1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/08939d4fd9aa49db9086bf21f65091d1 2023-04-20 20:56:11,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ee415c946ab145299215622f3d3f3926 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ee415c946ab145299215622f3d3f3926 2023-04-20 20:56:11,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/915f511d5dcc49a68025cd1c54ac80d0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/915f511d5dcc49a68025cd1c54ac80d0 2023-04-20 20:56:11,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/70727f565cec4198b7dc82173b7b47fc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/70727f565cec4198b7dc82173b7b47fc 2023-04-20 20:56:11,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/35e94df2838d4e5180ef682d8c0e3ad9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/35e94df2838d4e5180ef682d8c0e3ad9 2023-04-20 20:56:11,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8a6cba927ae34afb82c6aa0c44cd8a73 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8a6cba927ae34afb82c6aa0c44cd8a73 2023-04-20 20:56:11,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/eb87cce4f270421f97aa62363ab786ab to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/eb87cce4f270421f97aa62363ab786ab 2023-04-20 20:56:11,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8931e39300254f91acda58dcb2304814 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/8931e39300254f91acda58dcb2304814 2023-04-20 20:56:11,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/6017f72182114dbb9e29f097597a14df to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/6017f72182114dbb9e29f097597a14df 2023-04-20 20:56:11,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2bbae44c59794d4b8f2ba6b1790710d8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/2bbae44c59794d4b8f2ba6b1790710d8 2023-04-20 20:56:11,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/49cf7bbca7a947d08b32a6cc2f53a473 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/49cf7bbca7a947d08b32a6cc2f53a473 2023-04-20 20:56:11,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/d9ba039616a84650a4319c0f4847ee17 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/d9ba039616a84650a4319c0f4847ee17 2023-04-20 20:56:11,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fb91de30ec6140f6874774a3c0911c78, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a7b07ffb4a034b4f9a31d1fa11f3413f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86df1595b92948b3ab966e47f2abfa4b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/1ee186de1ec745fab6de4e412689ea34, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fca170cb0b4a447b948571c236dcaf3a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/576f6a780dad4ead84ff81d518fd4922, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/33c14dd6ea8246adabd79bbdafa7e764, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/bc44adb7116b4c559047004684e08285, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6b993e7804cc4768b6acaabe968770ab, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4e458f4dbad8440fb065bcf269cb88d1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/8999ec63d2d74cafb57be2b4a74e5e65, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/712886058ef846e4b87c4d9577d9553d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/659780e686bc4742a587ef598b342275, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a32aad2dbe9f462ea43a603b4278f822, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/38f6cd5269454a5da79e63701f3fe9fe, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/73d3604b42de4a6f989ca65e7844f81c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/216b4dc275e445409a1c0efb5c369432, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c522b7879e544951a9ae2d48e2e72416, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/821dc1bc385048b69b9ec70ebc32b349, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2cf1c245a2ed4f8cac38068c65d87775, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6ff18b08a25a44039626aa59bab7e024, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d2a02cf52594401d807f5eed5988e91f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b4792c50fc174cda96f82e412d5c6468, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/089e2c90e15c4e1abe5003c4dd1ee379, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/5fcb26efda934ff68fa7514552f6157e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0ea9978d448a4f10bc4cc33f33a85343, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/55d1404fd6d9423ab6d74d65e5734517, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/682187ec20bf41f88b0c943d0afd56d9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b7c473f76d18449cb3a73a8288f2b8d3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/68a4f6fe6439407489a3ed1149d2d9db, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/44a7569825b448b486399335a4c8536f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/f1ef9033cd8447c591a4cda71dc98107, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0b93aec68a664718add8d08085de0d49, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/dcd775c97dbd465d91a079c18aecf31e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d47c3255f52441a0997ff51b90f1a61e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0134cede14a84bfe90455f20d5354437, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9bce135cf7af4a1eb0191ad6b827bcd6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/7f165168bda446ffb699fd8866c5c6cf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/ab716b61c1334855a6a1e85f67a50c70, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a6491e9846fc449c8bf282f561d514d2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/77831d6e882440e0a5714cae72faeb74, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2a888c51d17545d89decf59dc99e0b32, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/12ae8ee1d2554ebb963ac0ef8921d087, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/066c90d067f64b9aaa5e29c1b89a5dcf, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4479d4c676bf4a75a68755f715a1d2a5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/382995d49fc64808a65bd47289031531, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c43e0fa13a414572a472af2ab6fee0d4] to archive 2023-04-20 20:56:11,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:56:11,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fb91de30ec6140f6874774a3c0911c78 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fb91de30ec6140f6874774a3c0911c78 2023-04-20 20:56:11,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a7b07ffb4a034b4f9a31d1fa11f3413f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a7b07ffb4a034b4f9a31d1fa11f3413f 2023-04-20 20:56:11,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86df1595b92948b3ab966e47f2abfa4b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86df1595b92948b3ab966e47f2abfa4b 2023-04-20 20:56:11,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/1ee186de1ec745fab6de4e412689ea34 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/1ee186de1ec745fab6de4e412689ea34 2023-04-20 20:56:11,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fca170cb0b4a447b948571c236dcaf3a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/fca170cb0b4a447b948571c236dcaf3a 2023-04-20 20:56:11,381 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/576f6a780dad4ead84ff81d518fd4922 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/576f6a780dad4ead84ff81d518fd4922 2023-04-20 20:56:11,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/33c14dd6ea8246adabd79bbdafa7e764 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/33c14dd6ea8246adabd79bbdafa7e764 2023-04-20 20:56:11,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/bc44adb7116b4c559047004684e08285 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/bc44adb7116b4c559047004684e08285 2023-04-20 20:56:11,384 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6b993e7804cc4768b6acaabe968770ab to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6b993e7804cc4768b6acaabe968770ab 2023-04-20 20:56:11,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4e458f4dbad8440fb065bcf269cb88d1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4e458f4dbad8440fb065bcf269cb88d1 2023-04-20 20:56:11,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/8999ec63d2d74cafb57be2b4a74e5e65 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/8999ec63d2d74cafb57be2b4a74e5e65 2023-04-20 20:56:11,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/712886058ef846e4b87c4d9577d9553d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/712886058ef846e4b87c4d9577d9553d 2023-04-20 20:56:11,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/659780e686bc4742a587ef598b342275 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/659780e686bc4742a587ef598b342275 2023-04-20 20:56:11,389 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a32aad2dbe9f462ea43a603b4278f822 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a32aad2dbe9f462ea43a603b4278f822 2023-04-20 20:56:11,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/38f6cd5269454a5da79e63701f3fe9fe to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/38f6cd5269454a5da79e63701f3fe9fe 2023-04-20 20:56:11,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/73d3604b42de4a6f989ca65e7844f81c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/73d3604b42de4a6f989ca65e7844f81c 2023-04-20 20:56:11,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/216b4dc275e445409a1c0efb5c369432 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/216b4dc275e445409a1c0efb5c369432 2023-04-20 20:56:11,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c522b7879e544951a9ae2d48e2e72416 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c522b7879e544951a9ae2d48e2e72416 2023-04-20 20:56:11,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/821dc1bc385048b69b9ec70ebc32b349 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/821dc1bc385048b69b9ec70ebc32b349 2023-04-20 20:56:11,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2cf1c245a2ed4f8cac38068c65d87775 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2cf1c245a2ed4f8cac38068c65d87775 2023-04-20 20:56:11,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6ff18b08a25a44039626aa59bab7e024 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/6ff18b08a25a44039626aa59bab7e024 2023-04-20 20:56:11,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d2a02cf52594401d807f5eed5988e91f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d2a02cf52594401d807f5eed5988e91f 2023-04-20 20:56:11,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b4792c50fc174cda96f82e412d5c6468 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b4792c50fc174cda96f82e412d5c6468 2023-04-20 20:56:11,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/089e2c90e15c4e1abe5003c4dd1ee379 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/089e2c90e15c4e1abe5003c4dd1ee379 2023-04-20 20:56:11,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/5fcb26efda934ff68fa7514552f6157e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/5fcb26efda934ff68fa7514552f6157e 2023-04-20 20:56:11,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0ea9978d448a4f10bc4cc33f33a85343 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0ea9978d448a4f10bc4cc33f33a85343 2023-04-20 20:56:11,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/55d1404fd6d9423ab6d74d65e5734517 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/55d1404fd6d9423ab6d74d65e5734517 2023-04-20 20:56:11,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/682187ec20bf41f88b0c943d0afd56d9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/682187ec20bf41f88b0c943d0afd56d9 2023-04-20 20:56:11,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b7c473f76d18449cb3a73a8288f2b8d3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/b7c473f76d18449cb3a73a8288f2b8d3 2023-04-20 20:56:11,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/68a4f6fe6439407489a3ed1149d2d9db to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/68a4f6fe6439407489a3ed1149d2d9db 2023-04-20 20:56:11,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/44a7569825b448b486399335a4c8536f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/44a7569825b448b486399335a4c8536f 2023-04-20 20:56:11,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/f1ef9033cd8447c591a4cda71dc98107 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/f1ef9033cd8447c591a4cda71dc98107 2023-04-20 20:56:11,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0b93aec68a664718add8d08085de0d49 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0b93aec68a664718add8d08085de0d49 2023-04-20 20:56:11,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/dcd775c97dbd465d91a079c18aecf31e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/dcd775c97dbd465d91a079c18aecf31e 2023-04-20 20:56:11,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d47c3255f52441a0997ff51b90f1a61e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/d47c3255f52441a0997ff51b90f1a61e 2023-04-20 20:56:11,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0134cede14a84bfe90455f20d5354437 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/0134cede14a84bfe90455f20d5354437 2023-04-20 20:56:11,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9bce135cf7af4a1eb0191ad6b827bcd6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9bce135cf7af4a1eb0191ad6b827bcd6 2023-04-20 20:56:11,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/7f165168bda446ffb699fd8866c5c6cf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/7f165168bda446ffb699fd8866c5c6cf 2023-04-20 20:56:11,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/ab716b61c1334855a6a1e85f67a50c70 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/ab716b61c1334855a6a1e85f67a50c70 2023-04-20 20:56:11,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a6491e9846fc449c8bf282f561d514d2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/a6491e9846fc449c8bf282f561d514d2 2023-04-20 20:56:11,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/77831d6e882440e0a5714cae72faeb74 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/77831d6e882440e0a5714cae72faeb74 2023-04-20 20:56:11,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2a888c51d17545d89decf59dc99e0b32 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/2a888c51d17545d89decf59dc99e0b32 2023-04-20 20:56:11,415 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/12ae8ee1d2554ebb963ac0ef8921d087 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/12ae8ee1d2554ebb963ac0ef8921d087 2023-04-20 20:56:11,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/066c90d067f64b9aaa5e29c1b89a5dcf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/066c90d067f64b9aaa5e29c1b89a5dcf 2023-04-20 20:56:11,417 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4479d4c676bf4a75a68755f715a1d2a5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/4479d4c676bf4a75a68755f715a1d2a5 2023-04-20 20:56:11,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/382995d49fc64808a65bd47289031531 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/382995d49fc64808a65bd47289031531 2023-04-20 20:56:11,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c43e0fa13a414572a472af2ab6fee0d4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c43e0fa13a414572a472af2ab6fee0d4 2023-04-20 20:56:11,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/ef8d8897201145b099236b9265d9a28d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70cc2a822968454780e3c3c53f18eef3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a0ae507575974742918043a4ca5a68ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dca83cd15d2a45cface16d0248946f0d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/82b33407f2fb4c378e7871acf82bea5b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dadfc2bdd0564a33be2bdac1b7b75ec0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f332fb920e74b779a0caa96921c52c3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f6e0ab3d5812465aaac2f9414300bfc2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e99073d26bf34a119a3bd25d1e729c5e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/bacecb747a324d1a8231c0af521afa02, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d09515dacf904cd890b0821d0b172b4b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f47ce5f16094bb3a4cde85abf6625e3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f362317780d6489e93474bb3f394a9b9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/576da2d64ba349989478876ead3b433d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a8c6241040f446738765f5179ad909c8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3ce6a26eda374ad59a6bedc0315e6e52, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/834cb95492574e61bcdae4797940b377, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2ea16a7317644339b83817f8cb026ee1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/853b4bf3959e47afbaceeacba0feac26, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/9ef917660f2e4f5a91fd86ba85c08433, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d0ffc1c9357d47778b91725005460aba, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f768d9fafdf4499fa34812541b46cc50, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/777f45feb6a84a7dbda387eca49b1034, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6087fb919ba2459383f30884106acc3a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e13c43c565db460bb3b910a053f98462, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/379964679f26416fa270590997c478a2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/07725cd7eb0c4e229a0942d590a4425e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b20e295d3c17432fb3702a810dcd00a0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/4338575858e54808a61bfc48fc6a94f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70ded06e777e4a2ba65cfa95a8073d7f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7115d07c802e4f098e6ccd16f1f3d8ae, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2fa5725e699f430fa238825665a15613, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d536cb24cda64e789957f98f5de5b38e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/fffe3680b98d4eaea003f59094976c50, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/0117eb5432b447b083a0b10dd6e2024d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3794802455114bba9906f6009009fc05, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b8c4b3c7478a423481d365d6fce43aa9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/88c189b56fae4ae48b6bb2966c0bd880, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d545782f08c54a799ccb52e4411a45bc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2527df9b5a944ad990a6f753cb638d9d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/95226192a8074be1837e9499b32be940, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7b1bed549fa943839af431085061b6ba, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b390baf9060b45f29c31d8e0a5581578, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/20f7a85189064e8e894929d77d2ae2fb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e10717d2c6774736a239c5e343254429, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/348e926513c5433fbacb77ae90acddb8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/41188177cc9645c186aa89f4de7d9aaa] to archive 2023-04-20 20:56:11,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:56:11,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/ef8d8897201145b099236b9265d9a28d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/ef8d8897201145b099236b9265d9a28d 2023-04-20 20:56:11,424 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70cc2a822968454780e3c3c53f18eef3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70cc2a822968454780e3c3c53f18eef3 2023-04-20 20:56:11,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a0ae507575974742918043a4ca5a68ad to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a0ae507575974742918043a4ca5a68ad 2023-04-20 20:56:11,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dca83cd15d2a45cface16d0248946f0d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dca83cd15d2a45cface16d0248946f0d 2023-04-20 20:56:11,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/82b33407f2fb4c378e7871acf82bea5b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/82b33407f2fb4c378e7871acf82bea5b 2023-04-20 20:56:11,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dadfc2bdd0564a33be2bdac1b7b75ec0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/dadfc2bdd0564a33be2bdac1b7b75ec0 2023-04-20 20:56:11,428 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f332fb920e74b779a0caa96921c52c3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f332fb920e74b779a0caa96921c52c3 2023-04-20 20:56:11,429 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f6e0ab3d5812465aaac2f9414300bfc2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f6e0ab3d5812465aaac2f9414300bfc2 2023-04-20 20:56:11,430 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e99073d26bf34a119a3bd25d1e729c5e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e99073d26bf34a119a3bd25d1e729c5e 2023-04-20 20:56:11,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/bacecb747a324d1a8231c0af521afa02 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/bacecb747a324d1a8231c0af521afa02 2023-04-20 20:56:11,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d09515dacf904cd890b0821d0b172b4b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d09515dacf904cd890b0821d0b172b4b 2023-04-20 20:56:11,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f47ce5f16094bb3a4cde85abf6625e3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6f47ce5f16094bb3a4cde85abf6625e3 2023-04-20 20:56:11,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f362317780d6489e93474bb3f394a9b9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f362317780d6489e93474bb3f394a9b9 2023-04-20 20:56:11,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/576da2d64ba349989478876ead3b433d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/576da2d64ba349989478876ead3b433d 2023-04-20 20:56:11,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a8c6241040f446738765f5179ad909c8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/a8c6241040f446738765f5179ad909c8 2023-04-20 20:56:11,435 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3ce6a26eda374ad59a6bedc0315e6e52 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3ce6a26eda374ad59a6bedc0315e6e52 2023-04-20 20:56:11,436 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/834cb95492574e61bcdae4797940b377 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/834cb95492574e61bcdae4797940b377 2023-04-20 20:56:11,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2ea16a7317644339b83817f8cb026ee1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2ea16a7317644339b83817f8cb026ee1 2023-04-20 20:56:11,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/853b4bf3959e47afbaceeacba0feac26 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/853b4bf3959e47afbaceeacba0feac26 2023-04-20 20:56:11,438 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/9ef917660f2e4f5a91fd86ba85c08433 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/9ef917660f2e4f5a91fd86ba85c08433 2023-04-20 20:56:11,439 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d0ffc1c9357d47778b91725005460aba to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d0ffc1c9357d47778b91725005460aba 2023-04-20 20:56:11,440 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f768d9fafdf4499fa34812541b46cc50 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/f768d9fafdf4499fa34812541b46cc50 2023-04-20 20:56:11,440 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/777f45feb6a84a7dbda387eca49b1034 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/777f45feb6a84a7dbda387eca49b1034 2023-04-20 20:56:11,441 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6087fb919ba2459383f30884106acc3a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/6087fb919ba2459383f30884106acc3a 2023-04-20 20:56:11,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e13c43c565db460bb3b910a053f98462 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e13c43c565db460bb3b910a053f98462 2023-04-20 20:56:11,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/379964679f26416fa270590997c478a2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/379964679f26416fa270590997c478a2 2023-04-20 20:56:11,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/07725cd7eb0c4e229a0942d590a4425e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/07725cd7eb0c4e229a0942d590a4425e 2023-04-20 20:56:11,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b20e295d3c17432fb3702a810dcd00a0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b20e295d3c17432fb3702a810dcd00a0 2023-04-20 20:56:11,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/4338575858e54808a61bfc48fc6a94f7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/4338575858e54808a61bfc48fc6a94f7 2023-04-20 20:56:11,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70ded06e777e4a2ba65cfa95a8073d7f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/70ded06e777e4a2ba65cfa95a8073d7f 2023-04-20 20:56:11,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7115d07c802e4f098e6ccd16f1f3d8ae to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7115d07c802e4f098e6ccd16f1f3d8ae 2023-04-20 20:56:11,447 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2fa5725e699f430fa238825665a15613 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2fa5725e699f430fa238825665a15613 2023-04-20 20:56:11,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d536cb24cda64e789957f98f5de5b38e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d536cb24cda64e789957f98f5de5b38e 2023-04-20 20:56:11,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/fffe3680b98d4eaea003f59094976c50 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/fffe3680b98d4eaea003f59094976c50 2023-04-20 20:56:11,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/0117eb5432b447b083a0b10dd6e2024d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/0117eb5432b447b083a0b10dd6e2024d 2023-04-20 20:56:11,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3794802455114bba9906f6009009fc05 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3794802455114bba9906f6009009fc05 2023-04-20 20:56:11,451 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b8c4b3c7478a423481d365d6fce43aa9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b8c4b3c7478a423481d365d6fce43aa9 2023-04-20 20:56:11,451 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/88c189b56fae4ae48b6bb2966c0bd880 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/88c189b56fae4ae48b6bb2966c0bd880 2023-04-20 20:56:11,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d545782f08c54a799ccb52e4411a45bc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/d545782f08c54a799ccb52e4411a45bc 2023-04-20 20:56:11,453 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2527df9b5a944ad990a6f753cb638d9d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/2527df9b5a944ad990a6f753cb638d9d 2023-04-20 20:56:11,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/95226192a8074be1837e9499b32be940 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/95226192a8074be1837e9499b32be940 2023-04-20 20:56:11,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7b1bed549fa943839af431085061b6ba to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/7b1bed549fa943839af431085061b6ba 2023-04-20 20:56:11,455 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b390baf9060b45f29c31d8e0a5581578 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/b390baf9060b45f29c31d8e0a5581578 2023-04-20 20:56:11,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/20f7a85189064e8e894929d77d2ae2fb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/20f7a85189064e8e894929d77d2ae2fb 2023-04-20 20:56:11,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e10717d2c6774736a239c5e343254429 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/e10717d2c6774736a239c5e343254429 2023-04-20 20:56:11,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/348e926513c5433fbacb77ae90acddb8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/348e926513c5433fbacb77ae90acddb8 2023-04-20 20:56:11,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/41188177cc9645c186aa89f4de7d9aaa to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/41188177cc9645c186aa89f4de7d9aaa 2023-04-20 20:56:11,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/recovered.edits/1039.seqid, newMaxSeqId=1039, maxSeqId=1 2023-04-20 20:56:11,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd. 2023-04-20 20:56:11,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8552cf5289bf62dd18e69f1f78b0b5fd: 2023-04-20 20:56:11,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:11,464 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=8552cf5289bf62dd18e69f1f78b0b5fd, regionState=CLOSED 2023-04-20 20:56:11,465 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024171464"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024171464"}]},"ts":"1682024171464"} 2023-04-20 20:56:11,466 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-20 20:56:11,466 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 8552cf5289bf62dd18e69f1f78b0b5fd, server=jenkins-hbase4.apache.org,39223,1682024020002 in 2.2030 sec 2023-04-20 20:56:11,467 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-20 20:56:11,467 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8552cf5289bf62dd18e69f1f78b0b5fd, UNASSIGN in 2.2140 sec 2023-04-20 20:56:11,468 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024171468"}]},"ts":"1682024171468"} 2023-04-20 20:56:11,469 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 20:56:11,470 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 20:56:11,471 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2260 sec 2023-04-20 20:56:13,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 20:56:13,354 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-20 20:56:13,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-20 20:56:13,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:13,356 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:13,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-20 20:56:13,357 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:13,360 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:13,362 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/recovered.edits] 2023-04-20 20:56:13,364 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/250022f28ed3428d8e04141e61403d9e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/250022f28ed3428d8e04141e61403d9e 2023-04-20 20:56:13,365 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4af59abec87b454096fe95d98f2900c0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/4af59abec87b454096fe95d98f2900c0 2023-04-20 20:56:13,366 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/e767ad35eea94b19810b218bf62281f8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/e767ad35eea94b19810b218bf62281f8 2023-04-20 20:56:13,367 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ff40de839c204b31a488e70c277222a4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/A/ff40de839c204b31a488e70c277222a4 2023-04-20 20:56:13,369 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86e84da0fb1b4b77baa10b80a4774527 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/86e84da0fb1b4b77baa10b80a4774527 2023-04-20 20:56:13,370 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9168ffdcfa1a46e5a0c1d6a65cf8b9ff to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/9168ffdcfa1a46e5a0c1d6a65cf8b9ff 2023-04-20 20:56:13,371 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/af40726cf7be43609d6a995924a5a27f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/af40726cf7be43609d6a995924a5a27f 2023-04-20 20:56:13,372 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c1353808cf684f3e9b384dc29352fe8f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/B/c1353808cf684f3e9b384dc29352fe8f 2023-04-20 20:56:13,373 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/1fa665a1eeb644ebbb784655361e26d7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/1fa665a1eeb644ebbb784655361e26d7 2023-04-20 20:56:13,374 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3e7c328536554ffea532a611aa4037be to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/3e7c328536554ffea532a611aa4037be 2023-04-20 20:56:13,375 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/879b53d5ab024a0fb303b756b7bdb5de to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/879b53d5ab024a0fb303b756b7bdb5de 2023-04-20 20:56:13,376 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/be982a1f0d30486dacf1f64745152161 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/C/be982a1f0d30486dacf1f64745152161 2023-04-20 20:56:13,378 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/recovered.edits/1039.seqid to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd/recovered.edits/1039.seqid 2023-04-20 20:56:13,378 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/8552cf5289bf62dd18e69f1f78b0b5fd 2023-04-20 20:56:13,379 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:56:13,380 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:13,381 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 20:56:13,383 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 20:56:13,384 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:13,384 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 20:56:13,384 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682024173384"}]},"ts":"9223372036854775807"} 2023-04-20 20:56:13,386 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 20:56:13,386 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8552cf5289bf62dd18e69f1f78b0b5fd, NAME => 'TestAcidGuarantees,,1682024147354.8552cf5289bf62dd18e69f1f78b0b5fd.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 20:56:13,386 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 20:56:13,386 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682024173386"}]},"ts":"9223372036854775807"} 2023-04-20 20:56:13,388 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 20:56:13,390 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:13,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-04-20 20:56:13,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-20 20:56:13,458 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-20 20:56:13,470 INFO [Listener at localhost/45619] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 233), OpenFileDescriptor=609 (was 551) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 409), ProcessCount=169 (was 169), AvailableMemoryMB=9060 (was 9184) 2023-04-20 20:56:13,482 INFO [Listener at localhost/45619] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=609, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=169, AvailableMemoryMB=9060 2023-04-20 20:56:13,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:56:13,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:56:13,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:13,486 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 20:56:13,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-20 20:56:13,487 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 20:56:13,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 20:56:13,488 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:13,489 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 empty. 2023-04-20 20:56:13,489 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:13,489 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:56:13,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 20:56:13,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 20:56:13,898 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:56:13,900 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 819071c32df35da7c00fe2675f028bf5, NAME => 'TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp 2023-04-20 20:56:14,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 20:56:14,305 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:56:14,305 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 819071c32df35da7c00fe2675f028bf5, disabling compactions & flushes 2023-04-20 20:56:14,305 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,305 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,305 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. after waiting 0 ms 2023-04-20 20:56:14,305 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,305 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,305 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:14,307 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 20:56:14,308 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024174308"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024174308"}]},"ts":"1682024174308"} 2023-04-20 20:56:14,309 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 20:56:14,309 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 20:56:14,310 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024174310"}]},"ts":"1682024174310"} 2023-04-20 20:56:14,310 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 20:56:14,313 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, ASSIGN}] 2023-04-20 20:56:14,314 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, ASSIGN 2023-04-20 20:56:14,315 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=false 2023-04-20 20:56:14,466 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:14,466 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024174466"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024174466"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024174466"}]},"ts":"1682024174466"} 2023-04-20 20:56:14,468 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:56:14,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 20:56:14,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 819071c32df35da7c00fe2675f028bf5, NAME => 'TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:56:14,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:56:14,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,624 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,625 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A 2023-04-20 20:56:14,625 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A 2023-04-20 20:56:14,626 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:56:14,626 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 819071c32df35da7c00fe2675f028bf5 columnFamilyName A 2023-04-20 20:56:14,626 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(310): Store=819071c32df35da7c00fe2675f028bf5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:56:14,626 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,627 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B 2023-04-20 20:56:14,627 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B 2023-04-20 20:56:14,627 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:56:14,628 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 819071c32df35da7c00fe2675f028bf5 columnFamilyName B 2023-04-20 20:56:14,628 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(310): Store=819071c32df35da7c00fe2675f028bf5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:56:14,628 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,629 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C 2023-04-20 20:56:14,629 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C 2023-04-20 20:56:14,629 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:56:14,629 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 819071c32df35da7c00fe2675f028bf5 columnFamilyName C 2023-04-20 20:56:14,630 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(310): Store=819071c32df35da7c00fe2675f028bf5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:56:14,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:56:14,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:14,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 20:56:14,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 819071c32df35da7c00fe2675f028bf5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69897423, jitterRate=0.041552767157554626}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:56:14,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:14,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., pid=56, masterSystemTime=1682024174619 2023-04-20 20:56:14,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:14,637 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:14,637 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024174637"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024174637"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024174637"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024174637"}]},"ts":"1682024174637"} 2023-04-20 20:56:14,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-20 20:56:14,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 in 170 msec 2023-04-20 20:56:14,640 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-20 20:56:14,641 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, ASSIGN in 326 msec 2023-04-20 20:56:14,641 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 20:56:14,641 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024174641"}]},"ts":"1682024174641"} 2023-04-20 20:56:14,642 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 20:56:14,644 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 20:56:14,645 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1600 sec 2023-04-20 20:56:15,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 20:56:15,592 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-20 20:56:15,593 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:15,599 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:15,600 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:15,602 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:15,603 DEBUG [Listener at localhost/45619] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 20:56:15,605 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33590, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 20:56:15,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 20:56:15,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 20:56:15,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:16,017 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 20:56:16,018 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 20:56:16,018 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 20:56:16,019 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-20 20:56:16,022 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, REOPEN/MOVE}] 2023-04-20 20:56:16,022 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, REOPEN/MOVE 2023-04-20 20:56:16,023 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,023 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024176023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024176023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024176023"}]},"ts":"1682024176023"} 2023-04-20 20:56:16,024 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:56:16,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 819071c32df35da7c00fe2675f028bf5, disabling compactions & flushes 2023-04-20 20:56:16,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. after waiting 0 ms 2023-04-20 20:56:16,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-20 20:56:16,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:16,180 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 819071c32df35da7c00fe2675f028bf5 to self. 2023-04-20 20:56:16,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,182 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=CLOSED 2023-04-20 20:56:16,182 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024176182"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024176182"}]},"ts":"1682024176182"} 2023-04-20 20:56:16,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-20 20:56:16,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 in 159 msec 2023-04-20 20:56:16,184 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,39223,1682024020002; forceNewPlan=false, retain=true 2023-04-20 20:56:16,334 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,335 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024176334"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024176334"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024176334"}]},"ts":"1682024176334"} 2023-04-20 20:56:16,336 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:56:16,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 819071c32df35da7c00fe2675f028bf5, NAME => 'TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.', STARTKEY => '', ENDKEY => ''} 2023-04-20 20:56:16,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 20:56:16,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,491 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,492 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A 2023-04-20 20:56:16,492 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A 2023-04-20 20:56:16,493 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:56:16,493 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 819071c32df35da7c00fe2675f028bf5 columnFamilyName A 2023-04-20 20:56:16,494 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(310): Store=819071c32df35da7c00fe2675f028bf5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:56:16,494 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,495 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B 2023-04-20 20:56:16,495 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B 2023-04-20 20:56:16,495 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:56:16,495 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 819071c32df35da7c00fe2675f028bf5 columnFamilyName B 2023-04-20 20:56:16,495 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(310): Store=819071c32df35da7c00fe2675f028bf5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:56:16,495 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,496 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C 2023-04-20 20:56:16,496 DEBUG [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C 2023-04-20 20:56:16,496 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 20:56:16,496 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 819071c32df35da7c00fe2675f028bf5 columnFamilyName C 2023-04-20 20:56:16,497 INFO [StoreOpener-819071c32df35da7c00fe2675f028bf5-1] regionserver.HStore(310): Store=819071c32df35da7c00fe2675f028bf5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 20:56:16,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 20:56:16,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 819071c32df35da7c00fe2675f028bf5; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66760688, jitterRate=-0.0051882266998291016}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 20:56:16,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:16,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., pid=61, masterSystemTime=1682024176487 2023-04-20 20:56:16,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,503 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,503 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024176503"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682024176503"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682024176503"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682024176503"}]},"ts":"1682024176503"} 2023-04-20 20:56:16,505 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-20 20:56:16,505 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 in 168 msec 2023-04-20 20:56:16,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-20 20:56:16,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, REOPEN/MOVE in 483 msec 2023-04-20 20:56:16,508 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-20 20:56:16,508 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-20 20:56:16,510 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-20 20:56:16,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-20 20:56:16,511 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,516 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,517 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,522 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,523 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,526 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,527 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,531 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,532 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,535 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,536 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,542 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,543 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,546 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,547 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,553 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,554 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,563 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,563 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(139): Connect 0x0eedd898 to 127.0.0.1:62055 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 20:56:16,569 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@308b3f39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 20:56:16,578 DEBUG [hconnection-0x7081921f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,580 DEBUG [hconnection-0x4f9c4082-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,580 DEBUG [hconnection-0x45d29560-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,580 DEBUG [hconnection-0x7c8ddd25-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,581 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,586 DEBUG [hconnection-0x6919ca06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,586 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,588 DEBUG [hconnection-0xe934dec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,592 DEBUG [hconnection-0x4e4cd235-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,592 DEBUG [hconnection-0x20d17a1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,593 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59472, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,594 DEBUG [hconnection-0x8e4904-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,594 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,594 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59520, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,593 DEBUG [hconnection-0x3f807647-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 20:56:16,594 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,596 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,601 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,601 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,603 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59566, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 20:56:16,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:16,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:16,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:56:16,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:16,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:16,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:16,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:16,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:16,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:16,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:16,619 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:16,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:16,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:16,619 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:16,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,620 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,621 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:16,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,622 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:16,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:16,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:16,623 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,623 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:16,623 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:16,623 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:16,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:16,626 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,626 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:16,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d6bbc[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:16,627 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:16,628 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,629 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:16,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:16,629 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:16,629 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,629 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:16,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:16,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:16,630 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,630 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:16,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:16,632 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:16,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:16,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:16,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:16,632 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:16,634 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,634 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,634 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:16,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:16,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:16,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,638 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,639 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:16,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:16,641 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:16,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:16,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:16,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:16,641 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,642 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,642 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:16,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:16,642 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024236671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024236672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024236672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024236672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024236674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,742 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:16,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:16,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:16,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:16,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:16,746 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:16,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:16,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:16,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:16,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,747 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:16,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:16,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:16,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:16,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:16,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:16,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:16,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:16,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:16,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:16,750 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:16,752 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6449a455[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:16,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:16,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,756 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,756 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:16,760 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:16,764 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:16,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:16,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:16,764 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:16,764 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:16,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,766 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:16,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,766 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,768 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,773 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:16,773 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:16,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:16,773 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,773 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:16,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:16,774 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,773 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:16,774 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,774 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,774 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,774 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:16,774 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:16,774 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024236775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024236775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024236776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024236776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024236777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,874 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:16,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:16,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:16,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:16,878 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:16,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:16,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:16,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:16,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:16,881 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:16,881 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,881 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:16,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:16,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:16,882 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,882 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:16,883 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:16,883 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:16,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,885 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:16,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ee6764e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:16,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:16,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:16,887 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:16,887 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,887 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:16,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:16,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:16,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:16,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:16,902 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:16,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:16,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:16,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:16,902 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:16,909 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,909 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:16,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,909 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:16,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:16,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:16,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:16,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:16,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:16,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:16,913 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:16,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:16,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:16,913 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:16,914 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:16,914 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:16,914 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:16,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:16,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024236978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024236979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024236979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,980 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 20:56:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024236979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024236979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,014 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,021 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,021 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,022 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,022 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,022 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,023 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,023 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,026 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,026 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,026 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:17,026 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:17,027 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:17,027 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,027 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:17,027 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:17,029 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,030 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:17,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5f8262[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:17,031 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:17,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,032 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420456ef784a2254031b5f95f5048f9997e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420456ef784a2254031b5f95f5048f9997e 2023-04-20 20:56:17,033 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,034 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:17,034 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:17,034 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,034 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:17,034 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:17,038 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:17,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:17,042 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:17,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:17,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:17,042 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:17,042 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:17,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,044 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,044 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:17,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:17,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,050 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:17,051 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:17,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:17,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,151 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,156 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,158 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,158 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,160 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:17,164 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:17,166 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:17,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119326fe[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:17,168 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:17,168 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,170 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:17,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:17,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:17,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:17,173 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:17,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,173 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:17,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:17,175 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:17,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:17,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:17,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:17,175 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:17,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,178 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,179 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:17,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,187 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:17,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,187 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:17,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:17,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024237281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024237283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024237283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024237283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024237284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,287 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,290 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,295 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,295 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,295 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:17,295 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:17,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38515395[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:17,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:17,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,301 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:17,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:17,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:17,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:17,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:17,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:17,305 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:17,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:17,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:17,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:17,305 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:17,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:17,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:17,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:17,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,313 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:17,313 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:17,413 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,416 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,421 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,421 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,421 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:17,421 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:17,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:17,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:17,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@259806d7[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:17,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:17,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,426 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:17,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:17,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:17,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:17,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:17,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:17,431 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:17,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:17,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:17,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:17,431 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:17,433 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,433 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,433 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,438 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:17,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:17,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,439 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:17,439 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:17,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,446 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/f80b87db520b445e891f3edb91f672f3 2023-04-20 20:56:17,539 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,542 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:17,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:17,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:17,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:17,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:17,549 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:17,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2455cc94[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:17,550 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:17,550 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,552 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,552 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:17,552 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:17,553 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:17,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:17,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:17,559 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:17,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:17,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:17,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:17,559 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:17,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,563 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,566 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:17,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:17,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:17,566 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:17,566 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:17,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,567 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,567 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,567 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,667 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,670 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,670 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,670 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,670 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,670 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,671 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,672 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,672 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,672 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,672 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:17,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:17,679 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,680 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:17,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cf9ba36[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:17,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:17,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,684 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:17,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:17,686 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:17,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:17,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:17,689 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:17,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:17,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:17,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:17,689 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:17,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,692 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,695 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:17,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,695 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,695 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:17,695 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:17,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024237783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024237787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024237787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024237788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024237788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,796 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,799 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,802 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,802 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,805 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,805 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,806 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,806 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,806 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:17,806 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:17,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8689c7[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:17,809 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:17,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,811 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:17,811 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,811 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:17,812 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:17,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:17,815 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:17,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:17,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:17,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:17,815 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:17,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:17,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:17,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:17,823 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,827 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:17,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:17,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:17,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:17,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:17,827 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:17,827 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:17,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:17,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/3672022998644506991fab3b90063485 2023-04-20 20:56:17,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/9d3abe677ed54ec2a9ea0e9f19794626 2023-04-20 20:56:17,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/f80b87db520b445e891f3edb91f672f3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f80b87db520b445e891f3edb91f672f3 2023-04-20 20:56:17,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f80b87db520b445e891f3edb91f672f3, entries=150, sequenceid=16, filesize=25.1 K 2023-04-20 20:56:17,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/3672022998644506991fab3b90063485 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3672022998644506991fab3b90063485 2023-04-20 20:56:17,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3672022998644506991fab3b90063485, entries=150, sequenceid=16, filesize=11.6 K 2023-04-20 20:56:17,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/9d3abe677ed54ec2a9ea0e9f19794626 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9d3abe677ed54ec2a9ea0e9f19794626 2023-04-20 20:56:17,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9d3abe677ed54ec2a9ea0e9f19794626, entries=150, sequenceid=16, filesize=11.6 K 2023-04-20 20:56:17,889 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 819071c32df35da7c00fe2675f028bf5 in 1276ms, sequenceid=16, compaction requested=false 2023-04-20 20:56:17,889 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-20 20:56:17,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:17,928 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:17,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:17,931 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:17,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:17,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:17,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,933 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:17,933 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:17,933 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:17,933 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:17,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:17,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:17,935 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:17,935 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:17,935 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 20:56:17,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:17,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:17,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:17,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:17,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:17,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:17,961 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042096a96c35b08e4e208e10a06265e64951 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042096a96c35b08e4e208e10a06265e64951 2023-04-20 20:56:17,971 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/37704a6c3d6a4e37a1e05edb02d49950 2023-04-20 20:56:18,005 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2cdb3409d3d64ebba9701ce1b01d40c7 2023-04-20 20:56:18,017 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/3bad02064bd54d8bb4075fec95ab5832 2023-04-20 20:56:18,021 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/37704a6c3d6a4e37a1e05edb02d49950 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/37704a6c3d6a4e37a1e05edb02d49950 2023-04-20 20:56:18,025 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/37704a6c3d6a4e37a1e05edb02d49950, entries=150, sequenceid=36, filesize=25.1 K 2023-04-20 20:56:18,025 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2cdb3409d3d64ebba9701ce1b01d40c7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2cdb3409d3d64ebba9701ce1b01d40c7 2023-04-20 20:56:18,028 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2cdb3409d3d64ebba9701ce1b01d40c7, entries=150, sequenceid=36, filesize=11.6 K 2023-04-20 20:56:18,029 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/3bad02064bd54d8bb4075fec95ab5832 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3bad02064bd54d8bb4075fec95ab5832 2023-04-20 20:56:18,032 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3bad02064bd54d8bb4075fec95ab5832, entries=150, sequenceid=36, filesize=11.6 K 2023-04-20 20:56:18,033 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 819071c32df35da7c00fe2675f028bf5 in 98ms, sequenceid=36, compaction requested=false 2023-04-20 20:56:18,034 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:18,034 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,035 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:18,035 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:18,035 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,035 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:18,035 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,036 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:18,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b4d8a7e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:18,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:18,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,040 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:18,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:18,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:18,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:18,045 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:18,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:18,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:18,045 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:18,045 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:18,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,047 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:18,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:18,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,052 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:18,052 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:18,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:18,152 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:18,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:18,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:18,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:18,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:18,157 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:18,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:18,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:18,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,159 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,160 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,160 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:18,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:18,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:18,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:18,162 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,162 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:18,162 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:18,162 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:18,164 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,164 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:18,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f6c8ef[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:18,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:18,166 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,169 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:18,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,170 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:18,170 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:18,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,171 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:18,174 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:18,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:18,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:18,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:18,174 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:18,176 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,176 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,177 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,178 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:18,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:18,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:18,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,182 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:18,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,182 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:18,183 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,183 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,283 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:18,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:18,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:18,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:18,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:18,286 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:18,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:18,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:18,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,287 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,288 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,288 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,288 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,289 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:18,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:18,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:18,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:18,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:18,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:18,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:18,290 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,290 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:18,290 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:18,290 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:18,292 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103fe4ed[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:18,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:18,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,296 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:18,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:18,301 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:18,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:18,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:18,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:18,301 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:18,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,304 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,304 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:18,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:18,309 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:18,309 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:18,409 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:18,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:18,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:18,413 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:18,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:18,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:18,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,415 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:18,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:18,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:18,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:18,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:18,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5d8f9c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:18,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:18,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:18,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:18,427 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:18,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:18,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:18,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:18,428 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:18,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,430 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:18,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:18,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,435 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:18,435 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:18,536 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:18,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:18,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:18,539 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:18,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:18,539 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:18,540 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,540 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,541 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,541 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,541 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:18,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:18,543 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,543 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:18,543 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:18,543 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:18,545 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dd4ecd3[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:18,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:18,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:18,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:18,549 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,549 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:18,551 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,551 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,552 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:18,554 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:18,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:18,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:18,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:18,554 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:18,558 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,558 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,558 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,560 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,562 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:18,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,563 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,563 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,563 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:18,563 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,663 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:18,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:18,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:18,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:18,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:18,667 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:18,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:18,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:18,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,668 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,669 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:18,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:18,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:18,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:18,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,671 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:18,672 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,672 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:18,672 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:18,672 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:18,674 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,675 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,675 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:18,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa2f836[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:18,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:18,676 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,677 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:18,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,678 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:18,679 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:18,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:18,682 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:18,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:18,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:18,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:18,682 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:18,684 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,684 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,684 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:18,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:18,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:18,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:18,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:18,690 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:18,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,690 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:18,791 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:18,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:18,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:18,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:18,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:18,794 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:18,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:18,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:18,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,796 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:18,796 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,797 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,797 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:18,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:18,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:18,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:18,798 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:18,798 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:18,799 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-20 20:56:18,799 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:18,799 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:18,800 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:18,800 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:18,800 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:18,800 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:18,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:18,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:18,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682024178822, max timestamp=1682024178827Num uniques 859; 2023-04-20 20:56:18,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682024178822, max timestamp=1682024178827Num uniques 859; 2023-04-20 20:56:18,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:18,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:18,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682024178822, max timestamp=1682024178827Num uniques 859; 2023-04-20 20:56:18,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:18,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024178844, max timestamp=1682024178846Num uniques 1118; 2023-04-20 20:56:18,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024178844, max timestamp=1682024178846Num uniques 1118; 2023-04-20 20:56:18,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024178844, max timestamp=1682024178846Num uniques 1118; 2023-04-20 20:56:18,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-20 20:56:18,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-20 20:56:18,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-20 20:56:18,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682024178857, max timestamp=1682024178863Num uniques 1077; 2023-04-20 20:56:18,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682024178857, max timestamp=1682024178863Num uniques 1077; 2023-04-20 20:56:18,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-20 20:56:18,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-20 20:56:18,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682024178857, max timestamp=1682024178863Num uniques 1077; 2023-04-20 20:56:18,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-20 20:56:18,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024178872, max timestamp=1682024178879Num uniques 1136; 2023-04-20 20:56:18,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-20 20:56:18,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024178872, max timestamp=1682024178879Num uniques 1136; 2023-04-20 20:56:18,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-20 20:56:18,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024178872, max timestamp=1682024178879Num uniques 1136; 2023-04-20 20:56:18,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-20 20:56:18,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682024178883, max timestamp=1682024178894Num uniques 1095; 2023-04-20 20:56:18,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-20 20:56:18,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682024178883, max timestamp=1682024178894Num uniques 1095; 2023-04-20 20:56:18,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-20 20:56:18,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682024178883, max timestamp=1682024178894Num uniques 1095; 2023-04-20 20:56:18,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-20 20:56:18,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024178904, max timestamp=1682024178910Num uniques 1104; 2023-04-20 20:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=13920, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-20 20:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024178904, max timestamp=1682024178910Num uniques 1104; 2023-04-20 20:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=13920, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-20 20:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024178904, max timestamp=1682024178910Num uniques 1104; 2023-04-20 20:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=13920, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-20 20:56:18,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024238925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:18,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682024178920, max timestamp=1682024178927Num uniques 1163; 2023-04-20 20:56:18,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682024178920, max timestamp=1682024178927Num uniques 1163; 2023-04-20 20:56:18,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=12038, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-20 20:56:18,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12037, totalHeapSize=27592, min timestamp=1682024178920, max timestamp=1682024178927Num uniques 1163; 2023-04-20 20:56:18,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=12038, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-20 20:56:18,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53257, new segment data size=12037, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-20 20:56:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024178939, max timestamp=1682024178945Num uniques 1122; 2023-04-20 20:56:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-20 20:56:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024178939, max timestamp=1682024178945Num uniques 1122; 2023-04-20 20:56:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-20 20:56:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024178939, max timestamp=1682024178945Num uniques 1122; 2023-04-20 20:56:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-20 20:56:18,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024178962, max timestamp=1682024178967Num uniques 1131; 2023-04-20 20:56:18,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-20 20:56:18,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024178962, max timestamp=1682024178967Num uniques 1131; 2023-04-20 20:56:18,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-20 20:56:18,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682024178962, max timestamp=1682024178967Num uniques 1131; 2023-04-20 20:56:18,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-20 20:56:19,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682024179024, max timestamp=1682024179027Num uniques 1090; 2023-04-20 20:56:19,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-20 20:56:19,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682024179024, max timestamp=1682024179027Num uniques 1090; 2023-04-20 20:56:19,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-20 20:56:19,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682024179024, max timestamp=1682024179027Num uniques 1090; 2023-04-20 20:56:19,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-20 20:56:19,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024179037, max timestamp=1682024179046Num uniques 1049; 2023-04-20 20:56:19,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:56:19,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024179037, max timestamp=1682024179046Num uniques 1049; 2023-04-20 20:56:19,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:56:19,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024179037, max timestamp=1682024179046Num uniques 1049; 2023-04-20 20:56:19,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:56:19,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682024179058, max timestamp=1682024179064Num uniques 1108; 2023-04-20 20:56:19,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-20 20:56:19,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682024179058, max timestamp=1682024179064Num uniques 1108; 2023-04-20 20:56:19,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-20 20:56:19,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024179058, max timestamp=1682024179064Num uniques 1108; 2023-04-20 20:56:19,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=6870, suffix heap size=143200,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=150 2023-04-20 20:56:19,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024179074, max timestamp=1682024179086Num uniques 1009; 2023-04-20 20:56:19,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=9932, suffix heap size=132904,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=217 2023-04-20 20:56:19,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024179074, max timestamp=1682024179086Num uniques 1067; 2023-04-20 20:56:19,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-20 20:56:19,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1682024179074, max timestamp=1682024179086Num uniques 1067; 2023-04-20 20:56:19,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-20 20:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1682024179116, max timestamp=1682024179121Num uniques 1076; 2023-04-20 20:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1682024179116, max timestamp=1682024179121Num uniques 1126; 2023-04-20 20:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-04-20 20:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-20 20:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1682024179116, max timestamp=1682024179121Num uniques 1076; 2023-04-20 20:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-04-20 20:56:19,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024179131, max timestamp=1682024179137Num uniques 1185; 2023-04-20 20:56:19,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-04-20 20:56:19,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024179131, max timestamp=1682024179137Num uniques 1185; 2023-04-20 20:56:19,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-04-20 20:56:19,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024179131, max timestamp=1682024179137Num uniques 1185; 2023-04-20 20:56:19,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-04-20 20:56:19,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1682024179145, max timestamp=1682024179152Num uniques 1094; 2023-04-20 20:56:19,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=15754, suffix heap size=141744,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=344 2023-04-20 20:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1682024179145, max timestamp=1682024179152Num uniques 1094; 2023-04-20 20:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=15754, suffix heap size=141744,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=344 2023-04-20 20:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1682024179145, max timestamp=1682024179152Num uniques 1094; 2023-04-20 20:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=15754, suffix heap size=141744,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=344 2023-04-20 20:56:19,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024239191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024239192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024239192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682024179183, max timestamp=1682024179195Num uniques 1203; 2023-04-20 20:56:19,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=7005, suffix heap size=153080,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=153 2023-04-20 20:56:19,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682024179183, max timestamp=1682024179195Num uniques 1203; 2023-04-20 20:56:19,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=7005, suffix heap size=153080,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=153 2023-04-20 20:56:19,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682024179183, max timestamp=1682024179195Num uniques 1203; 2023-04-20 20:56:19,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=7005, suffix heap size=153080,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=153 2023-04-20 20:56:19,212 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042092564d2e04a741b2858acb99e6903e30 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042092564d2e04a741b2858acb99e6903e30 2023-04-20 20:56:19,218 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=47, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/8c440ff8a713478291f693ce0a510c1a 2023-04-20 20:56:19,228 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/15c574e6f2314102b604a1a150eccb5e 2023-04-20 20:56:19,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682024179210, max timestamp=1682024179228Num uniques 1012; 2023-04-20 20:56:19,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682024179210, max timestamp=1682024179228Num uniques 1012; 2023-04-20 20:56:19,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682024179210, max timestamp=1682024179228Num uniques 1012; 2023-04-20 20:56:19,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-20 20:56:19,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-20 20:56:19,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-20 20:56:19,241 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/8197825cfa4542c8b866769249d5474e 2023-04-20 20:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/8c440ff8a713478291f693ce0a510c1a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8c440ff8a713478291f693ce0a510c1a 2023-04-20 20:56:19,248 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8c440ff8a713478291f693ce0a510c1a, entries=150, sequenceid=47, filesize=25.1 K 2023-04-20 20:56:19,249 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/15c574e6f2314102b604a1a150eccb5e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/15c574e6f2314102b604a1a150eccb5e 2023-04-20 20:56:19,252 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/15c574e6f2314102b604a1a150eccb5e, entries=100, sequenceid=47, filesize=9.3 K 2023-04-20 20:56:19,253 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/8197825cfa4542c8b866769249d5474e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8197825cfa4542c8b866769249d5474e 2023-04-20 20:56:19,256 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8197825cfa4542c8b866769249d5474e, entries=100, sequenceid=47, filesize=9.3 K 2023-04-20 20:56:19,257 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=134.18 KB/137400 for 819071c32df35da7c00fe2675f028bf5 in 459ms, sequenceid=47, compaction requested=true 2023-04-20 20:56:19,257 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:19,257 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:19,258 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,258 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:19,259 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=140.89 KB heapSize=399.94 KB 2023-04-20 20:56:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:19,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:56:19,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:19,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:56:19,260 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:19,260 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 20:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:19,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d56103[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:19,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:19,263 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,264 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:19,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:19,264 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:19,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:19,266 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:19,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:19,269 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:19,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:19,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:19,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:19,269 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:19,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024239268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024239270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,274 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d145101028e54256849e5e0250bfb1cf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d145101028e54256849e5e0250bfb1cf 2023-04-20 20:56:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:19,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:19,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:19,277 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:19,277 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:19,277 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,284 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/8af9b807bbf0415c8f0943fe0515401b 2023-04-20 20:56:19,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/f68b9643729c419ba13c87f8c46dba89 2023-04-20 20:56:19,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024239299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024239299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024239300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/de905861e6ac43f9aab887fbcfaa6f5b 2023-04-20 20:56:19,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/8af9b807bbf0415c8f0943fe0515401b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8af9b807bbf0415c8f0943fe0515401b 2023-04-20 20:56:19,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8af9b807bbf0415c8f0943fe0515401b, entries=150, sequenceid=377, filesize=25.4 K 2023-04-20 20:56:19,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/f68b9643729c419ba13c87f8c46dba89 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f68b9643729c419ba13c87f8c46dba89 2023-04-20 20:56:19,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f68b9643729c419ba13c87f8c46dba89, entries=150, sequenceid=377, filesize=11.9 K 2023-04-20 20:56:19,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/de905861e6ac43f9aab887fbcfaa6f5b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/de905861e6ac43f9aab887fbcfaa6f5b 2023-04-20 20:56:19,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/de905861e6ac43f9aab887fbcfaa6f5b, entries=150, sequenceid=377, filesize=11.9 K 2023-04-20 20:56:19,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.52 KB/430608, currentSize=33.54 KB/34350 for 819071c32df35da7c00fe2675f028bf5 in 68ms, sequenceid=377, compaction requested=true 2023-04-20 20:56:19,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:19,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:19,327 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:19,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:19,327 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:19,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:19,328 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103280 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:19,328 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/A is initiating minor compaction (all files) 2023-04-20 20:56:19,328 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45500 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:19,328 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/A in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,328 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:19,328 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f80b87db520b445e891f3edb91f672f3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/37704a6c3d6a4e37a1e05edb02d49950, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8c440ff8a713478291f693ce0a510c1a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8af9b807bbf0415c8f0943fe0515401b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=100.9 K 2023-04-20 20:56:19,328 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,328 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3672022998644506991fab3b90063485, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2cdb3409d3d64ebba9701ce1b01d40c7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/15c574e6f2314102b604a1a150eccb5e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f68b9643729c419ba13c87f8c46dba89] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=44.4 K 2023-04-20 20:56:19,328 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f80b87db520b445e891f3edb91f672f3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682024176608 2023-04-20 20:56:19,329 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3672022998644506991fab3b90063485, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682024176608 2023-04-20 20:56:19,329 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 37704a6c3d6a4e37a1e05edb02d49950, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682024176670 2023-04-20 20:56:19,329 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2cdb3409d3d64ebba9701ce1b01d40c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682024176670 2023-04-20 20:56:19,329 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8c440ff8a713478291f693ce0a510c1a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024178797 2023-04-20 20:56:19,329 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 15c574e6f2314102b604a1a150eccb5e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024178797 2023-04-20 20:56:19,329 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 8af9b807bbf0415c8f0943fe0515401b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682024179242 2023-04-20 20:56:19,329 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f68b9643729c419ba13c87f8c46dba89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682024179242 2023-04-20 20:56:19,341 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#551 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:19,365 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#A#compaction#552 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:19,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:19,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:19,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:19,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:19,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:19,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:19,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:19,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:19,377 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:19,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:19,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:19,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:19,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:19,381 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:19,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:19,381 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:19,382 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,382 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,383 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,384 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:19,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:19,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:19,400 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:19,400 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:19,400 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,401 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:19,401 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:19,401 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:19,403 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:19,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53222f4a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:19,406 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:19,406 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,407 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,407 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,408 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:19,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:19,412 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:19,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:19,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:19,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:19,412 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:19,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024179409, max timestamp=1682024179415Num uniques 859; 2023-04-20 20:56:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024179409, max timestamp=1682024179415Num uniques 859; 2023-04-20 20:56:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024179409, max timestamp=1682024179415Num uniques 859; 2023-04-20 20:56:19,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:19,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,423 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:19,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:19,423 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,424 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:19,424 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:19,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024239443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024239444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024239504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024239506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024239506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,524 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:19,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:19,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:19,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:19,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:19,527 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:19,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:19,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:19,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,528 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,529 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,530 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:19,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:19,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:19,531 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,531 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:19,532 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:19,532 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:19,534 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599af389[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:19,536 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:19,536 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,538 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:19,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,538 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:19,539 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:19,542 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:19,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:19,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:19,542 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:19,542 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:19,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,543 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:19,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:19,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:19,547 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,548 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:19,548 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:19,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024239546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024239548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,648 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:19,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:19,652 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:19,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:19,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:19,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,653 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,655 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,655 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:19,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:19,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:19,657 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,658 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:19,658 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:19,658 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:19,658 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:19,659 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,660 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:19,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2484c3c4[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:19,662 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:19,662 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,663 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:19,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,664 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:19,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:19,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,665 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,665 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:19,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:19,668 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:19,668 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:19,668 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:19,668 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:19,668 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:19,670 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,670 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,670 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,671 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,672 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,674 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:19,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,675 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,675 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:19,675 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:19,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:19,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024239750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024239751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,776 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:19,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:19,777 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/f80ea50a0dbc4ffcb320db6ad965a398 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f80ea50a0dbc4ffcb320db6ad965a398 2023-04-20 20:56:19,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:19,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:19,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:19,779 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:19,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:19,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:19,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,781 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,781 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:19,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,782 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into f80ea50a0dbc4ffcb320db6ad965a398(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:19,782 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:19,782 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=12, startTime=1682024179327; duration=0sec 2023-04-20 20:56:19,783 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:19,783 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:19,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:19,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:19,785 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45500 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:19,785 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,786 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/C is initiating minor compaction (all files) 2023-04-20 20:56:19,786 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:19,786 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/C in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,786 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:19,786 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,786 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9d3abe677ed54ec2a9ea0e9f19794626, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3bad02064bd54d8bb4075fec95ab5832, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8197825cfa4542c8b866769249d5474e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/de905861e6ac43f9aab887fbcfaa6f5b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=44.4 K 2023-04-20 20:56:19,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:19,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:19,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,786 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9d3abe677ed54ec2a9ea0e9f19794626, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682024176608 2023-04-20 20:56:19,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:19,786 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:19,787 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3bad02064bd54d8bb4075fec95ab5832, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682024176670 2023-04-20 20:56:19,787 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8197825cfa4542c8b866769249d5474e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682024178797 2023-04-20 20:56:19,787 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting de905861e6ac43f9aab887fbcfaa6f5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682024179242 2023-04-20 20:56:19,788 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,788 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@af7732e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:19,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:19,790 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,792 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:19,793 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#C#compaction#554 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:19,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:19,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:19,794 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:19,795 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,795 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,799 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420b14ab756707244ddab48b8bfb8a48ef6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b14ab756707244ddab48b8bfb8a48ef6 2023-04-20 20:56:19,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:19,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:19,805 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:19,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:19,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:19,806 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:19,806 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:19,807 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,808 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,809 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,811 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,814 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/13645249b80a49be8e14ba788d36f468 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/13645249b80a49be8e14ba788d36f468 2023-04-20 20:56:19,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024239812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024239812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:19,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:19,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:19,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024239812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,818 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:19,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,818 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:19,819 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:19,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,819 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,821 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/C of 819071c32df35da7c00fe2675f028bf5 into 13645249b80a49be8e14ba788d36f468(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:19,821 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:19,821 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/C, priority=12, startTime=1682024179327; duration=0sec 2023-04-20 20:56:19,821 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:19,919 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:19,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:19,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:19,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:19,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:19,922 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:19,923 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,923 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:19,923 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:19,923 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,923 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,924 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,925 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,925 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:19,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:19,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:19,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:19,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,927 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:19,927 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:19,927 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:19,929 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:19,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486d5545[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:19,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:19,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,934 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:19,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:19,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:19,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:19,936 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:19,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:19,939 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:19,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:19,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:19,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:19,939 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:19,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:19,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:19,942 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,943 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,946 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:19,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:19,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:19,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:19,946 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:19,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:19,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:19,946 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:19,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,047 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,051 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:20,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024240054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:20,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59482 deadline: 1682024240056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,057 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,057 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,059 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4665f17a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,063 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,063 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,064 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,064 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,068 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,068 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,072 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,073 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,075 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,075 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,075 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,075 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,075 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:20,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,075 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,076 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,076 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,076 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,076 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,076 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,076 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,176 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,180 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,180 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,184 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,186 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@447d9091[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,188 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,190 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,190 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,195 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,195 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,196 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,196 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,196 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,197 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,199 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,202 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,202 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:20,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,223 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/c212933e32b341f881275d3fdd424064 2023-04-20 20:56:20,302 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,305 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:20,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,310 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,310 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,310 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,311 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,314 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a2f533[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,316 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,318 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,318 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,318 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,319 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024240319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,322 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,322 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,322 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,325 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024240322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024240323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,328 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,329 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,329 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,329 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,329 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:20,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:20,429 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,432 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,434 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,435 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,436 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,436 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,436 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,436 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,436 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,438 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,438 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c751bf[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,442 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,442 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,442 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,443 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,448 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,448 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,448 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,448 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,448 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,449 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,449 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,450 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,451 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:20,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,454 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,454 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,454 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:20,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,554 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,558 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:20,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024240560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:20,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:20,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024240561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,564 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,564 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,564 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,564 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,566 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,566 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,566 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,566 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec0941c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,568 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,569 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,569 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,571 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,571 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,574 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,574 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,574 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,574 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,574 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,577 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,578 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,581 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,581 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,581 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,582 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,582 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,582 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,582 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,582 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,582 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,582 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,582 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:20,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,615 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/f91fd0af652449a4a2698be348448d1a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f91fd0af652449a4a2698be348448d1a 2023-04-20 20:56:20,620 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/A of 819071c32df35da7c00fe2675f028bf5 into f91fd0af652449a4a2698be348448d1a(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-20 20:56:20,620 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:20,620 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/A, priority=12, startTime=1682024179327; duration=1sec 2023-04-20 20:56:20,620 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:20,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2991e7d490dd4b4ea6bc3fbd660c4576 2023-04-20 20:56:20,682 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,685 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,685 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,687 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,687 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,687 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,689 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,690 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1691624d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,692 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,692 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,696 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,697 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,697 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,698 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,698 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,698 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,700 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,700 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,701 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,702 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,710 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,711 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,711 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:20,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:20,811 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,815 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,817 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,817 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,817 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,817 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:20,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,818 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,819 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,819 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,819 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,820 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747806c1[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,822 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,823 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,829 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,829 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,829 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,829 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,829 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,830 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,830 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,830 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,831 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,832 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,835 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,835 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,835 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,835 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:20,936 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:20,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:20,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:20,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:20,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:20,939 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:20,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:20,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:20,940 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,940 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,941 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:20,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:20,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:20,943 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,943 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:20,943 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:20,943 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:20,943 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:20,946 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:20,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195ffb58[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:20,947 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:20,948 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,950 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:20,950 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:20,950 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:20,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,950 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:20,951 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:20,954 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:20,954 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:20,954 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:20,954 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:20,954 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:20,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:20,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,956 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,957 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:20,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,962 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:20,962 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:20,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:20,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:20,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:20,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:20,962 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:20,962 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,962 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:20,962 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:20,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:20,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:20,962 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:20,962 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:21,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/55b7cbb792554dd8bb2a3110b17a8a5a 2023-04-20 20:56:21,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/c212933e32b341f881275d3fdd424064 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c212933e32b341f881275d3fdd424064 2023-04-20 20:56:21,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c212933e32b341f881275d3fdd424064, entries=200, sequenceid=387, filesize=32.3 K 2023-04-20 20:56:21,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2991e7d490dd4b4ea6bc3fbd660c4576 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2991e7d490dd4b4ea6bc3fbd660c4576 2023-04-20 20:56:21,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2991e7d490dd4b4ea6bc3fbd660c4576, entries=150, sequenceid=387, filesize=11.9 K 2023-04-20 20:56:21,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/55b7cbb792554dd8bb2a3110b17a8a5a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55b7cbb792554dd8bb2a3110b17a8a5a 2023-04-20 20:56:21,062 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:21,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:21,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:21,065 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:21,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:21,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:21,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:21,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:21,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:21,067 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:21,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:21,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,067 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:21,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:21,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:21,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55b7cbb792554dd8bb2a3110b17a8a5a, entries=150, sequenceid=387, filesize=11.9 K 2023-04-20 20:56:21,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 819071c32df35da7c00fe2675f028bf5 in 1693ms, sequenceid=387, compaction requested=false 2023-04-20 20:56:21,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:21,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:21,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:21,072 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:21,072 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:21,072 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:21,072 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:21,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:21,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:21,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024241335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024241336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024241338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024241440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024241440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024241442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,488 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420685b00de68214d88942db59ff8f3f273 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420685b00de68214d88942db59ff8f3f273 2023-04-20 20:56:21,496 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=427, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/cf6ca19fec014dbd9e2b0b01f1da8794 2023-04-20 20:56:21,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024241569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024241573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024241643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024241644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024241644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,907 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/5dfc7bb73912404e806270e7be19f727 2023-04-20 20:56:21,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024241948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024241949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:21,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024241950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:21,992 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 20:56:22,318 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/39fd6f6e8cb141599b7a565ff0d1b0ad 2023-04-20 20:56:22,322 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/cf6ca19fec014dbd9e2b0b01f1da8794 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/cf6ca19fec014dbd9e2b0b01f1da8794 2023-04-20 20:56:22,325 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/cf6ca19fec014dbd9e2b0b01f1da8794, entries=150, sequenceid=427, filesize=25.4 K 2023-04-20 20:56:22,326 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/5dfc7bb73912404e806270e7be19f727 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5dfc7bb73912404e806270e7be19f727 2023-04-20 20:56:22,329 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5dfc7bb73912404e806270e7be19f727, entries=150, sequenceid=427, filesize=11.9 K 2023-04-20 20:56:22,330 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/39fd6f6e8cb141599b7a565ff0d1b0ad as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/39fd6f6e8cb141599b7a565ff0d1b0ad 2023-04-20 20:56:22,333 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/39fd6f6e8cb141599b7a565ff0d1b0ad, entries=150, sequenceid=427, filesize=11.9 K 2023-04-20 20:56:22,333 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 819071c32df35da7c00fe2675f028bf5 in 1261ms, sequenceid=427, compaction requested=true 2023-04-20 20:56:22,333 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:22,334 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:22,334 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:22,336 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,336 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,336 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,336 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2385fc1b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:22,338 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:22,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,340 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,340 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:22,340 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:22,340 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:22,340 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,340 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:22,348 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:22,351 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:22,351 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:22,351 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:22,351 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:22,351 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:22,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,352 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,354 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,355 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:22,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,358 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,359 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:22,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,359 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:22,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 20:56:22,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:22,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:22,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:22,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:22,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:22,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:22,459 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:22,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:22,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:22,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:22,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:22,463 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:22,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:22,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:22,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,465 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,465 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:22,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:22,467 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,467 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:22,467 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:22,467 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:22,470 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,470 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9b712c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:22,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:22,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:22,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:22,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:22,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:22,477 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:22,477 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,477 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,477 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:22,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:22,480 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:22,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:22,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:22,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:22,480 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:22,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,483 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024242482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682024182462, max timestamp=1682024182484Num uniques 859; 2023-04-20 20:56:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682024182462, max timestamp=1682024182484Num uniques 859; 2023-04-20 20:56:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682024182462, max timestamp=1682024182484Num uniques 859; 2023-04-20 20:56:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:22,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:22,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,487 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:22,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:22,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,487 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:22,488 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:22,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,488 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:22,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024242510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024242511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,588 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:22,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:22,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:22,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024242588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:22,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:22,591 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:22,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:22,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:22,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,593 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,594 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,594 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:22,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:22,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:22,595 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,595 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:22,595 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:22,595 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:22,597 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,597 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae12630[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:22,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:22,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,601 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,601 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,601 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:22,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:22,601 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:22,602 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:22,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:22,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,604 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:22,606 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:22,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:22,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:22,607 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:22,607 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:22,608 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,608 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,609 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,609 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,611 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,613 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:22,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:22,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,614 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:22,614 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:22,613 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:22,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024242614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024242614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,714 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:22,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:22,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:22,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:22,719 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:22,719 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,719 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:22,719 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:22,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,723 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,723 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:22,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:22,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:22,725 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,726 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:22,726 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:22,726 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:22,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:22,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:22,726 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:22,728 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,728 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,729 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,729 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c02890f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:22,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:22,732 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,734 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:22,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:22,734 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:22,734 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:22,735 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,736 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:22,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:22,738 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:22,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:22,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:22,738 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:22,738 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:22,740 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,740 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,740 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,740 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,742 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:22,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,748 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,748 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,748 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:22,749 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:22,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024242793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024242817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:22,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024242818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,849 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:22,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:22,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:22,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:22,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:22,853 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:22,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:22,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:22,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,858 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:22,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:22,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:22,860 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:22,860 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,860 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:22,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:22,868 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,868 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:22,868 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:22,868 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:22,869 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:22,870 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:22,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@606bd5b0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:22,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:22,873 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,875 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:22,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:22,877 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:22,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,877 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304207f03b0b8dc294354834e311425561b32 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207f03b0b8dc294354834e311425561b32 2023-04-20 20:56:22,878 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:22,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:22,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:22,882 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:22,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:22,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:22,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:22,882 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:22,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:22,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:22,886 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,887 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:22,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:22,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:22,892 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,892 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:22,892 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:22,892 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:22,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:22,898 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=438, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/f7bf718834814c538e6e02175eab76a8 2023-04-20 20:56:22,992 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:22,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:22,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:22,997 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:22,997 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:22,997 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:22,997 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:23,002 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,002 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,004 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,004 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:23,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:23,014 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,014 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:23,014 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:23,015 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:23,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:23,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:23,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:23,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,017 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,017 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:23,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cca80[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:23,023 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:23,023 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,025 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:23,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:23,026 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:23,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:23,032 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:23,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:23,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:23,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:23,032 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:23,034 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,035 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,035 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,035 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,039 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:23,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:23,041 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:23,041 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,042 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,043 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024243098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024243121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024243123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,142 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:23,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:23,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:23,146 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:23,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:23,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:23,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,147 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,147 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,148 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,148 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:23,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:23,149 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,149 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:23,150 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:23,150 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:23,151 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64facb5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:23,154 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:23,154 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:23,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:23,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:23,156 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:23,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:23,162 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,162 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:23,165 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:23,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:23,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:23,165 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:23,165 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:23,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,167 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,167 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,169 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,171 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:23,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:23,171 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,172 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,172 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:23,172 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:23,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:23,272 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:23,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:23,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:23,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:23,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:23,277 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:23,277 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,277 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:23,277 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:23,277 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,277 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,279 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,279 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,279 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:23,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:23,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:23,281 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,281 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:23,281 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:23,281 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:23,282 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7efd1782[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:23,285 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:23,285 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,286 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:23,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:23,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:23,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:23,288 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:23,291 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:23,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:23,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:23,291 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:23,291 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:23,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,293 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,294 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:23,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:23,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,298 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:23,298 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:23,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:23,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/07439551088440eda12c2e2893e2099c 2023-04-20 20:56:23,398 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:23,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:23,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:23,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:23,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:23,402 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:23,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:23,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:23,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,403 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,404 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,404 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,404 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,404 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:23,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:23,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:23,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:23,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:23,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:23,409 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:23,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40859bb8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:23,411 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:23,412 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,413 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:23,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,414 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,414 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:23,417 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:23,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:23,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:23,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:23,417 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:23,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,419 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,422 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:23,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,425 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,425 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:23,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:23,425 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:23,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,525 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:23,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:23,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:23,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:23,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:23,529 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:23,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:23,529 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:23,530 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,530 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,531 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,531 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,532 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,532 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:23,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:23,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:23,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:23,544 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:23,544 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:23,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ae11e2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:23,549 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:23,549 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,550 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:23,550 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,550 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:23,553 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,553 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:23,553 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,553 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:23,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:23,556 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:23,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:23,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:23,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:23,556 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:23,557 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,558 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,558 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,564 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:23,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,565 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:23,565 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:23,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:23,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024243583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,588 DEBUG [Thread-4008] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:56:23,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59522 deadline: 1682024243590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,592 DEBUG [Thread-4002] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., hostname=jenkins-hbase4.apache.org,39223,1682024020002, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 20:56:23,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024243603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59566 deadline: 1682024243627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:23,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59536 deadline: 1682024243628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,665 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:23,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:23,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:23,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:23,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:23,669 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:23,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:23,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:23,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,674 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,674 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,675 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:23,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:23,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:23,676 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,676 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:23,676 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:23,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:23,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:23,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:23,678 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,678 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,679 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:23,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ff3b26[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:23,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:23,680 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:23,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:23,683 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:23,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,683 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:23,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:23,686 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:23,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:23,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:23,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:23,686 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:23,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:23,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:23,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,693 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:23,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:23,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:23,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:23,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:23,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:23,693 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:23,693 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:23,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/7903923367ee43009a14cf8a230f0286 2023-04-20 20:56:23,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/f7bf718834814c538e6e02175eab76a8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f7bf718834814c538e6e02175eab76a8 2023-04-20 20:56:23,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f7bf718834814c538e6e02175eab76a8, entries=150, sequenceid=438, filesize=25.4 K 2023-04-20 20:56:23,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/07439551088440eda12c2e2893e2099c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/07439551088440eda12c2e2893e2099c 2023-04-20 20:56:23,739 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/07439551088440eda12c2e2893e2099c, entries=100, sequenceid=438, filesize=9.5 K 2023-04-20 20:56:23,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/7903923367ee43009a14cf8a230f0286 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/7903923367ee43009a14cf8a230f0286 2023-04-20 20:56:23,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/7903923367ee43009a14cf8a230f0286, entries=100, sequenceid=438, filesize=9.5 K 2023-04-20 20:56:23,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 819071c32df35da7c00fe2675f028bf5 in 1290ms, sequenceid=438, compaction requested=true 2023-04-20 20:56:23,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:23,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:23,746 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:23,746 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:23,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:23,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:23,751 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:23,751 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46438 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:23,751 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/A is initiating minor compaction (all files) 2023-04-20 20:56:23,751 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:23,751 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/A in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,751 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,751 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f91fd0af652449a4a2698be348448d1a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c212933e32b341f881275d3fdd424064, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/cf6ca19fec014dbd9e2b0b01f1da8794, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f7bf718834814c538e6e02175eab76a8] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=108.7 K 2023-04-20 20:56:23,751 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f80ea50a0dbc4ffcb320db6ad965a398, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2991e7d490dd4b4ea6bc3fbd660c4576, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5dfc7bb73912404e806270e7be19f727, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/07439551088440eda12c2e2893e2099c] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=45.3 K 2023-04-20 20:56:23,752 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f91fd0af652449a4a2698be348448d1a, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682024179242 2023-04-20 20:56:23,752 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f80ea50a0dbc4ffcb320db6ad965a398, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682024179242 2023-04-20 20:56:23,753 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting c212933e32b341f881275d3fdd424064, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682024179264 2023-04-20 20:56:23,753 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2991e7d490dd4b4ea6bc3fbd660c4576, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682024179264 2023-04-20 20:56:23,753 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting cf6ca19fec014dbd9e2b0b01f1da8794, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682024179436 2023-04-20 20:56:23,753 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5dfc7bb73912404e806270e7be19f727, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682024179436 2023-04-20 20:56:23,754 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting f7bf718834814c538e6e02175eab76a8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024181335 2023-04-20 20:56:23,754 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 07439551088440eda12c2e2893e2099c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024181337 2023-04-20 20:56:23,762 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#563 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:23,766 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#A#compaction#564 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:23,785 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/f1099b3d23be4e9a97f940cd4645c02e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f1099b3d23be4e9a97f940cd4645c02e 2023-04-20 20:56:23,790 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into f1099b3d23be4e9a97f940cd4645c02e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:23,790 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:23,790 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=12, startTime=1682024183746; duration=0sec 2023-04-20 20:56:23,791 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:23,791 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:23,792 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46438 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:23,792 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/C is initiating minor compaction (all files) 2023-04-20 20:56:23,792 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/C in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,792 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/13645249b80a49be8e14ba788d36f468, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55b7cbb792554dd8bb2a3110b17a8a5a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/39fd6f6e8cb141599b7a565ff0d1b0ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/7903923367ee43009a14cf8a230f0286] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=45.3 K 2023-04-20 20:56:23,793 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 13645249b80a49be8e14ba788d36f468, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682024179242 2023-04-20 20:56:23,793 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 55b7cbb792554dd8bb2a3110b17a8a5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682024179264 2023-04-20 20:56:23,793 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 39fd6f6e8cb141599b7a565ff0d1b0ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1682024179436 2023-04-20 20:56:23,793 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:23,794 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 7903923367ee43009a14cf8a230f0286, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024181337 2023-04-20 20:56:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:23,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:23,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:23,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:23,798 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:23,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:23,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:23,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,798 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,799 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,799 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:23,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:23,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:23,800 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,800 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#C#compaction#565 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:23,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:23,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:23,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:23,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:23,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:23,802 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:23,816 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:23,816 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:23,816 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 20:56:23,816 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:23,816 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:23,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:23,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:23,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:23,817 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:23,831 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/8fc3e148d0a547a1b88ad290ac3e4428 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8fc3e148d0a547a1b88ad290ac3e4428 2023-04-20 20:56:23,834 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/C of 819071c32df35da7c00fe2675f028bf5 into 8fc3e148d0a547a1b88ad290ac3e4428(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:23,835 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:23,835 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/C, priority=12, startTime=1682024183746; duration=0sec 2023-04-20 20:56:23,835 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:23,840 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/e1af8b6aea44403bbf4b0f615707b0ad as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e1af8b6aea44403bbf4b0f615707b0ad 2023-04-20 20:56:23,845 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/A of 819071c32df35da7c00fe2675f028bf5 into e1af8b6aea44403bbf4b0f615707b0ad(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:23,845 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:23,845 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/A, priority=12, startTime=1682024183746; duration=0sec 2023-04-20 20:56:23,845 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:23,853 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420ce6599e61b4e4dc095eec60035e3310f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ce6599e61b4e4dc095eec60035e3310f 2023-04-20 20:56:23,858 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/9ee2c10afdf44993a717230dd70719ed 2023-04-20 20:56:23,868 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/906442be2899453e97e26258da556b71 2023-04-20 20:56:24,278 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/120fef011cb44229915a73de3d97a505 2023-04-20 20:56:24,282 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/9ee2c10afdf44993a717230dd70719ed as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/9ee2c10afdf44993a717230dd70719ed 2023-04-20 20:56:24,285 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/9ee2c10afdf44993a717230dd70719ed, entries=150, sequenceid=474, filesize=25.4 K 2023-04-20 20:56:24,286 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/906442be2899453e97e26258da556b71 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/906442be2899453e97e26258da556b71 2023-04-20 20:56:24,289 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/906442be2899453e97e26258da556b71, entries=150, sequenceid=474, filesize=11.9 K 2023-04-20 20:56:24,290 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/120fef011cb44229915a73de3d97a505 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/120fef011cb44229915a73de3d97a505 2023-04-20 20:56:24,293 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/120fef011cb44229915a73de3d97a505, entries=150, sequenceid=474, filesize=11.9 K 2023-04-20 20:56:24,294 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 819071c32df35da7c00fe2675f028bf5 in 478ms, sequenceid=474, compaction requested=false 2023-04-20 20:56:24,294 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:24,294 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:24,296 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ea876bc[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:24,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:24,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:24,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:24,301 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:24,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:24,302 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:24,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:24,305 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:24,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:24,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:24,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:24,305 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:24,307 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,307 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,307 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,311 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:24,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,312 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:24,312 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:24,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:24,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,312 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,412 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:24,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:24,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:24,416 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:24,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:24,416 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:24,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,417 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,418 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,418 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:24,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:24,420 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,420 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:24,421 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:24,421 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,422 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229c7c8a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:24,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:24,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,426 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:24,426 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,426 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:24,428 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:24,430 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:24,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:24,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:24,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:24,431 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:24,433 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,433 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,433 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,434 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:24,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:24,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,439 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:24,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,439 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,539 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:24,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:24,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:24,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:24,543 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:24,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:24,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:24,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,544 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:24,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:24,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:24,546 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:24,546 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:24,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:24,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:24,548 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ee583a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:24,550 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:24,550 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:24,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,554 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:24,556 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:24,558 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:24,558 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:24,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:24,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:24,559 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:24,560 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,560 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,560 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,561 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,562 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,564 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,565 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:24,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:24,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,565 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,564 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,565 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,565 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,565 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:24,565 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,565 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:24,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,665 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:24,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:24,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:24,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:24,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:24,669 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:24,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:24,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:24,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,672 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,672 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,672 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,672 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,672 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:24,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:24,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:24,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:24,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:24,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:24,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:24,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:24,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:24,674 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:24,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,674 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:24,675 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:24,675 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:24,675 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:24,677 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,677 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:24,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50697b2a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:24,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:24,679 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,680 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:24,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,681 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:24,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:24,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,682 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,682 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:24,686 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:24,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:24,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:24,686 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:24,686 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:24,688 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304204945030d552e4ea894e6728bf898a8ca to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304204945030d552e4ea894e6728bf898a8ca 2023-04-20 20:56:24,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,689 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,690 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,692 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,693 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=489, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/b8dae304f72043b1bfe0a8ba13a98100 2023-04-20 20:56:24,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,699 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:24,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:24,699 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,699 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:24,700 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:24,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:24,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59472 deadline: 1682024244703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024184692, max timestamp=1682024184704Num uniques 859; 2023-04-20 20:56:24,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:24,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024184692, max timestamp=1682024184704Num uniques 859; 2023-04-20 20:56:24,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:24,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024184692, max timestamp=1682024184704Num uniques 859; 2023-04-20 20:56:24,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:24,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024244745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:24,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024244746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,800 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:24,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:24,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:24,803 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:24,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:24,803 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:24,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:24,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024244806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,810 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:24,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:24,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:24,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:24,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d79ea0d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:24,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:24,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,817 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:24,817 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,817 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:24,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:24,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,819 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:24,821 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:24,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:24,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:24,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:24,821 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:24,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,822 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,823 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,824 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,826 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,827 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:24,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:24,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:24,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,827 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,827 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:24,827 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:24,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024244849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:24,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024244850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,927 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:24,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:24,931 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:24,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:24,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:24,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,933 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,933 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,934 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:24,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:24,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:24,935 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,935 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:24,935 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:24,936 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:24,937 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:24,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc0a504[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:24,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:24,939 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:24,940 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,940 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:24,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:24,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,945 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:24,948 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:24,948 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:24,948 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:24,948 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:24,948 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:24,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:24,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,950 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:24,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,951 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,954 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:24,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:24,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:24,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:24,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:24,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:24,955 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:24,955 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:24,960 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:24,960 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024245010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024245053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024245053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,060 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:25,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:25,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:25,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:25,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:25,064 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:25,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:25,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:25,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,065 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:25,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:25,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:25,067 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,068 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:25,068 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:25,068 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:25,075 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fec012e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:25,077 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:25,077 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,079 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:25,079 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,079 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:25,081 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:25,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:25,084 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:25,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:25,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:25,084 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:25,084 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:25,085 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,085 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,085 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,087 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,090 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,091 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:25,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:25,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,091 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,091 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,091 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,091 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,091 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:25,091 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:25,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/b534188b1d8a4686b4b4fe27e737e10b 2023-04-20 20:56:25,191 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:25,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:25,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:25,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:25,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:25,194 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:25,194 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,194 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:25,194 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:25,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,196 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,196 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,196 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,196 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:25,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:25,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:25,198 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,198 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:25,198 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:25,198 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:25,199 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,200 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,200 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c799d2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:25,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:25,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:25,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:25,202 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:25,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,202 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:25,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:25,207 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:25,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:25,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:25,207 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:25,207 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:25,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,209 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,209 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,210 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,213 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,212 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:25,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:25,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:25,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,213 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,213 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:25,213 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,313 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:25,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024245312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:25,317 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:25,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:25,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:25,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,317 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,319 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,319 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:25,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:25,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:25,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:25,321 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,324 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,324 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:25,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16fc22c5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:25,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:25,326 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:25,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:25,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,329 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:25,332 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:25,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:25,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:25,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:25,332 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:25,333 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,334 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,334 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:25,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:25,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,342 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,342 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:25,342 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024245356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024245357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,442 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:25,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:25,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:25,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:25,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:25,446 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:25,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:25,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:25,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,447 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,448 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,448 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:25,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:25,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:25,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:25,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:25,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:25,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:25,453 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:25,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f0d5d9b[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:25,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:25,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,456 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:25,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:25,459 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,459 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:25,478 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:25,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:25,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:25,478 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:25,478 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:25,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,480 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,480 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,482 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,484 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:25,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,484 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:25,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,484 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,484 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:25,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/1833c889288f4bdc9bde1a3874c71902 2023-04-20 20:56:25,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/b8dae304f72043b1bfe0a8ba13a98100 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/b8dae304f72043b1bfe0a8ba13a98100 2023-04-20 20:56:25,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/b8dae304f72043b1bfe0a8ba13a98100, entries=350, sequenceid=489, filesize=52.9 K 2023-04-20 20:56:25,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/b534188b1d8a4686b4b4fe27e737e10b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b534188b1d8a4686b4b4fe27e737e10b 2023-04-20 20:56:25,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b534188b1d8a4686b4b4fe27e737e10b, entries=150, sequenceid=489, filesize=11.9 K 2023-04-20 20:56:25,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/1833c889288f4bdc9bde1a3874c71902 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1833c889288f4bdc9bde1a3874c71902 2023-04-20 20:56:25,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1833c889288f4bdc9bde1a3874c71902, entries=150, sequenceid=489, filesize=11.9 K 2023-04-20 20:56:25,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 819071c32df35da7c00fe2675f028bf5 in 859ms, sequenceid=489, compaction requested=true 2023-04-20 20:56:25,531 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:25,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:25,531 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:25,531 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:25,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:25,531 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:25,532 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:25,532 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:25,532 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/A is initiating minor compaction (all files) 2023-04-20 20:56:25,532 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:25,532 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/A in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,532 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,532 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e1af8b6aea44403bbf4b0f615707b0ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/9ee2c10afdf44993a717230dd70719ed, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/b8dae304f72043b1bfe0a8ba13a98100] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=104.1 K 2023-04-20 20:56:25,532 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f1099b3d23be4e9a97f940cd4645c02e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/906442be2899453e97e26258da556b71, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b534188b1d8a4686b4b4fe27e737e10b] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=36.0 K 2023-04-20 20:56:25,533 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e1af8b6aea44403bbf4b0f615707b0ad, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024179436 2023-04-20 20:56:25,533 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting f1099b3d23be4e9a97f940cd4645c02e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024179436 2023-04-20 20:56:25,533 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 9ee2c10afdf44993a717230dd70719ed, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682024182506 2023-04-20 20:56:25,535 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting b8dae304f72043b1bfe0a8ba13a98100, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682024184612 2023-04-20 20:56:25,535 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 906442be2899453e97e26258da556b71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682024182506 2023-04-20 20:56:25,536 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b534188b1d8a4686b4b4fe27e737e10b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682024184655 2023-04-20 20:56:25,541 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#573 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:25,554 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#A#compaction#572 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:25,572 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/17689612029f4422be45fb733c230f84 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/17689612029f4422be45fb733c230f84 2023-04-20 20:56:25,576 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into 17689612029f4422be45fb733c230f84(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:25,576 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:25,576 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=13, startTime=1682024185531; duration=0sec 2023-04-20 20:56:25,576 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:25,576 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:25,577 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:25,577 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/C is initiating minor compaction (all files) 2023-04-20 20:56:25,577 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/C in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,578 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8fc3e148d0a547a1b88ad290ac3e4428, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/120fef011cb44229915a73de3d97a505, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1833c889288f4bdc9bde1a3874c71902] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=36.0 K 2023-04-20 20:56:25,578 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8fc3e148d0a547a1b88ad290ac3e4428, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682024179436 2023-04-20 20:56:25,578 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 120fef011cb44229915a73de3d97a505, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682024182506 2023-04-20 20:56:25,578 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 1833c889288f4bdc9bde1a3874c71902, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682024184655 2023-04-20 20:56:25,583 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#C#compaction#574 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:25,585 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:25,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:25,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:25,588 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:25,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:25,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:25,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,588 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,590 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,591 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:25,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:25,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:25,592 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:25,592 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:25,593 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 20:56:25,593 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:25,593 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:25,593 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:25,593 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:25,593 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:25,593 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:25,611 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420edfdf7c1b57742f281b8183c88d89773 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420edfdf7c1b57742f281b8183c88d89773 2023-04-20 20:56:25,617 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=525, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/7c1c2e7363a045e4b267c30323749141 2023-04-20 20:56:25,629 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/05b32cc6b31743ef94320ad751cb21ba 2023-04-20 20:56:25,639 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/97882247397144f78fb0fddf47800dd0 2023-04-20 20:56:25,642 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/7c1c2e7363a045e4b267c30323749141 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/7c1c2e7363a045e4b267c30323749141 2023-04-20 20:56:25,646 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/7c1c2e7363a045e4b267c30323749141, entries=150, sequenceid=525, filesize=25.4 K 2023-04-20 20:56:25,646 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/05b32cc6b31743ef94320ad751cb21ba as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/05b32cc6b31743ef94320ad751cb21ba 2023-04-20 20:56:25,649 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/05b32cc6b31743ef94320ad751cb21ba, entries=150, sequenceid=525, filesize=11.9 K 2023-04-20 20:56:25,650 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/97882247397144f78fb0fddf47800dd0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/97882247397144f78fb0fddf47800dd0 2023-04-20 20:56:25,654 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/97882247397144f78fb0fddf47800dd0, entries=150, sequenceid=525, filesize=11.9 K 2023-04-20 20:56:25,656 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 819071c32df35da7c00fe2675f028bf5 in 64ms, sequenceid=525, compaction requested=false 2023-04-20 20:56:25,656 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:25,656 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,656 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:25,659 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7097d825[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:25,661 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:25,661 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,662 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:25,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:25,662 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,662 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:25,662 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:25,664 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:25,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:25,667 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:25,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:25,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:25,667 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:25,667 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:25,668 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,668 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,668 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,669 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,670 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:25,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:25,673 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,673 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:25,673 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,674 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,674 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,774 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:25,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:25,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:25,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:25,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:25,778 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:25,778 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,778 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:25,778 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:25,778 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,778 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,779 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,779 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,780 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,780 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:25,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:25,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:25,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,781 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:25,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:25,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:25,782 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,782 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:25,782 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:25,782 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:25,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:25,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:25,782 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:25,783 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,783 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,784 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,784 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4826c6ec[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:25,785 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:25,786 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,787 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,787 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:25,787 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:25,787 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:25,787 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,787 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:25,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:25,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,789 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:25,789 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:25,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:25,792 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:25,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:25,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:25,792 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:25,792 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:25,793 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,794 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,794 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,795 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,801 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:25,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:25,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:25,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,801 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,802 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:25,802 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,802 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,802 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,802 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,802 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:25,867 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:25,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:25,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:25,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:25,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:25,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:25,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:25,882 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304201920e3d4b31a494c95e898d640c35029 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201920e3d4b31a494c95e898d640c35029 2023-04-20 20:56:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024185884, max timestamp=1682024185898Num uniques 859; 2023-04-20 20:56:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024185884, max timestamp=1682024185898Num uniques 859; 2023-04-20 20:56:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682024185884, max timestamp=1682024185898Num uniques 859; 2023-04-20 20:56:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:25,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 20:56:25,902 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:25,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:25,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:25,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:25,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:25,908 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:25,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:25,908 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:25,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,909 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,911 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,911 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:25,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:25,916 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:25,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024245922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024245922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:25,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:25,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:25,926 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:25,926 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:25,928 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024245924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,929 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:25,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbb2f27[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:25,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:25,931 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,932 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:25,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,932 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:25,934 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:25,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:25,937 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:25,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:25,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:25,937 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:25,937 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:25,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,940 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,941 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,942 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:25,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:25,944 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:25,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:25,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:25,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:25,944 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:25,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:25,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:25,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:25,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:25,945 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:25,945 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:25,945 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:25,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:25,995 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/e41a76bfc9564698859f170adb121e82 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e41a76bfc9564698859f170adb121e82 2023-04-20 20:56:26,000 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/C of 819071c32df35da7c00fe2675f028bf5 into e41a76bfc9564698859f170adb121e82(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:26,000 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:26,000 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/C, priority=13, startTime=1682024185531; duration=0sec 2023-04-20 20:56:26,000 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:26,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024246027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024246027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024246030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,045 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:26,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:26,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:26,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:26,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:26,048 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:26,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:26,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:26,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,049 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,050 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:26,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:26,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,051 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:26,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:26,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:26,052 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,052 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:26,052 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:26,052 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:26,054 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,054 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,055 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5abf4cce[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:26,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:26,056 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,059 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:26,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,059 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:26,061 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:26,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:26,064 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:26,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:26,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:26,064 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:26,064 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:26,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,065 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:26,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:26,066 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,068 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:26,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:26,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:26,071 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:26,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:26,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:26,070 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,071 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,071 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:26,071 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:26,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:26,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:26,171 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:26,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:26,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:26,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:26,174 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:26,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:26,174 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:26,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,175 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,176 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:26,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:26,176 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,177 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:26,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:26,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:26,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:26,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:26,178 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:26,178 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:26,180 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a7f14c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:26,181 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:26,181 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:26,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,182 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:26,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:26,184 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:26,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:26,186 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:26,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:26,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:26,186 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:26,186 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:26,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,188 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,189 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:26,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,192 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:26,192 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:26,192 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:26,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:26,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:26,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:26,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:26,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024246231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024246231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024246234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,292 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:26,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:26,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:26,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:26,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:26,296 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:26,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:26,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:26,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,296 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,300 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:26,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:26,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:26,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:26,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:26,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:26,302 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,302 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:26,302 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:26,302 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:26,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:26,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:26,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ac85492[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:26,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:26,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,314 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:26,314 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=536, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/6037f07d1d9e46fb8ee8a5ce9b870076 2023-04-20 20:56:26,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:26,320 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:26,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:26,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:26,320 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:26,320 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:26,322 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,322 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:26,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,322 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:26,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:26,323 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,324 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:26,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:26,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:26,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:26,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:26,328 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:26,328 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:26,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:26,328 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:26,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:26,339 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/db5328af4d6242c7b8bd5e31cbec4d44 2023-04-20 20:56:26,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/1bb29498372e454db538969c64356969 2023-04-20 20:56:26,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/6037f07d1d9e46fb8ee8a5ce9b870076 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6037f07d1d9e46fb8ee8a5ce9b870076 2023-04-20 20:56:26,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6037f07d1d9e46fb8ee8a5ce9b870076, entries=150, sequenceid=536, filesize=25.4 K 2023-04-20 20:56:26,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/db5328af4d6242c7b8bd5e31cbec4d44 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/db5328af4d6242c7b8bd5e31cbec4d44 2023-04-20 20:56:26,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/db5328af4d6242c7b8bd5e31cbec4d44, entries=150, sequenceid=536, filesize=11.9 K 2023-04-20 20:56:26,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/1bb29498372e454db538969c64356969 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1bb29498372e454db538969c64356969 2023-04-20 20:56:26,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1bb29498372e454db538969c64356969, entries=150, sequenceid=536, filesize=11.9 K 2023-04-20 20:56:26,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 819071c32df35da7c00fe2675f028bf5 in 498ms, sequenceid=536, compaction requested=true 2023-04-20 20:56:26,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:26,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:26,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:26,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 20:56:26,365 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-20 20:56:26,366 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 20:56:26,366 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 20:56:26,366 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. because compaction request was cancelled 2023-04-20 20:56:26,366 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:26,367 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:26,367 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:26,367 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,367 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/17689612029f4422be45fb733c230f84, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/05b32cc6b31743ef94320ad751cb21ba, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/db5328af4d6242c7b8bd5e31cbec4d44] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=36.1 K 2023-04-20 20:56:26,368 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 17689612029f4422be45fb733c230f84, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682024184655 2023-04-20 20:56:26,368 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 05b32cc6b31743ef94320ad751cb21ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1682024184710 2023-04-20 20:56:26,369 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting db5328af4d6242c7b8bd5e31cbec4d44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682024185840 2023-04-20 20:56:26,376 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#581 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:26,376 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/0cfeeb95d9634a4b993c0384568cfbe6 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0cfeeb95d9634a4b993c0384568cfbe6 2023-04-20 20:56:26,382 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/A of 819071c32df35da7c00fe2675f028bf5 into 0cfeeb95d9634a4b993c0384568cfbe6(size=25.8 K), total size for store is 76.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:26,382 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:26,382 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/A, priority=13, startTime=1682024185531; duration=0sec 2023-04-20 20:56:26,382 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:26,382 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:26,383 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:26,383 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/C is initiating minor compaction (all files) 2023-04-20 20:56:26,383 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/C in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,383 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e41a76bfc9564698859f170adb121e82, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/97882247397144f78fb0fddf47800dd0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1bb29498372e454db538969c64356969] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=36.1 K 2023-04-20 20:56:26,384 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e41a76bfc9564698859f170adb121e82, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682024184655 2023-04-20 20:56:26,384 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 97882247397144f78fb0fddf47800dd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1682024184710 2023-04-20 20:56:26,385 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 1bb29498372e454db538969c64356969, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682024185840 2023-04-20 20:56:26,385 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/830bc03e57244fe8bc3cb2ef162240b5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/830bc03e57244fe8bc3cb2ef162240b5 2023-04-20 20:56:26,389 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#C#compaction#582 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:26,396 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into 830bc03e57244fe8bc3cb2ef162240b5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:26,396 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:26,396 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=13, startTime=1682024186365; duration=0sec 2023-04-20 20:56:26,396 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:26,428 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:26,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:26,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:26,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:26,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:26,431 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:26,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:26,431 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:26,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,432 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,433 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:26,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:26,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:26,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:26,434 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:26,434 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:26,434 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:26,435 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-20 20:56:26,435 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:26,435 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:26,435 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:26,435 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:26,435 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:26,435 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:26,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:26,539 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:26,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024246546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024246546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024246548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024246649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024246651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024246653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,800 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/adb476adccf0456b8e8c0ac33a070fa5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/adb476adccf0456b8e8c0ac33a070fa5 2023-04-20 20:56:26,804 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/C of 819071c32df35da7c00fe2675f028bf5 into adb476adccf0456b8e8c0ac33a070fa5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:26,804 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:26,804 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/C, priority=13, startTime=1682024186365; duration=0sec 2023-04-20 20:56:26,804 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:26,849 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d05a85aa380d4576800efe021de93b65 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d05a85aa380d4576800efe021de93b65 2023-04-20 20:56:26,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024246854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024246855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:26,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:26,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024246858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024247158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024247159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024247162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,254 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=575, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/1683d40670104f5196fbf25657da4e3c 2023-04-20 20:56:27,262 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/4eb2780f57c845bba935398d8eeae271 2023-04-20 20:56:27,272 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/95edeb826ece4ce58b4e6275744e6416 2023-04-20 20:56:27,276 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/1683d40670104f5196fbf25657da4e3c as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/1683d40670104f5196fbf25657da4e3c 2023-04-20 20:56:27,279 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/1683d40670104f5196fbf25657da4e3c, entries=150, sequenceid=575, filesize=25.4 K 2023-04-20 20:56:27,280 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/4eb2780f57c845bba935398d8eeae271 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/4eb2780f57c845bba935398d8eeae271 2023-04-20 20:56:27,283 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/4eb2780f57c845bba935398d8eeae271, entries=150, sequenceid=575, filesize=11.9 K 2023-04-20 20:56:27,284 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/95edeb826ece4ce58b4e6275744e6416 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/95edeb826ece4ce58b4e6275744e6416 2023-04-20 20:56:27,288 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/95edeb826ece4ce58b4e6275744e6416, entries=150, sequenceid=575, filesize=11.9 K 2023-04-20 20:56:27,288 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 819071c32df35da7c00fe2675f028bf5 in 853ms, sequenceid=575, compaction requested=true 2023-04-20 20:56:27,288 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:27,288 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:27,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:27,288 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:27,290 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,290 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cec9cfb[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:27,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:27,293 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,294 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:27,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,295 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:27,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:27,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:27,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:27,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:27,300 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:27,300 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:27,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:27,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:27,301 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:27,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:27,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:27,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:27,304 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,305 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:27,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:27,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:27,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,308 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:27,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:27,309 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:27,309 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:27,309 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:27,309 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:27,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:27,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:27,409 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:27,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:27,413 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:27,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:27,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:27,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:27,413 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,414 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:27,415 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:27,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:27,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:27,415 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:27,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:27,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:27,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:27,417 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:27,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:27,417 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:27,417 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:27,418 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:27,418 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:27,418 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:27,418 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:27,418 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:27,418 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:27,434 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042096e23ccb6e024077931422eeb607696c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042096e23ccb6e024077931422eeb607696c 2023-04-20 20:56:27,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:27,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:27,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024187653, max timestamp=1682024187661Num uniques 859; 2023-04-20 20:56:27,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:27,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024187653, max timestamp=1682024187661Num uniques 859; 2023-04-20 20:56:27,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:27,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024187653, max timestamp=1682024187661Num uniques 859; 2023-04-20 20:56:27,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:27,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024187673, max timestamp=1682024187680Num uniques 1018; 2023-04-20 20:56:27,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:56:27,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024187673, max timestamp=1682024187680Num uniques 1018; 2023-04-20 20:56:27,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682024187673, max timestamp=1682024187680Num uniques 1018; 2023-04-20 20:56:27,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:56:27,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-20 20:56:27,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024247693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024247694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024247695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024247696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024247697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024247797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024247798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024247799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024247800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:27,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024247801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,840 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=586, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/6fb5ba04dd34496b9798d6cebea15d62 2023-04-20 20:56:27,858 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/3d6ad2996c5742e8b524370df9323bbe 2023-04-20 20:56:27,873 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/9e96caffcccd49cc8a3fb23eaa22f98b 2023-04-20 20:56:27,878 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/6fb5ba04dd34496b9798d6cebea15d62 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6fb5ba04dd34496b9798d6cebea15d62 2023-04-20 20:56:27,887 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6fb5ba04dd34496b9798d6cebea15d62, entries=150, sequenceid=586, filesize=25.4 K 2023-04-20 20:56:27,901 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/3d6ad2996c5742e8b524370df9323bbe as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3d6ad2996c5742e8b524370df9323bbe 2023-04-20 20:56:27,905 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3d6ad2996c5742e8b524370df9323bbe, entries=150, sequenceid=586, filesize=11.9 K 2023-04-20 20:56:27,905 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/9e96caffcccd49cc8a3fb23eaa22f98b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9e96caffcccd49cc8a3fb23eaa22f98b 2023-04-20 20:56:27,908 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9e96caffcccd49cc8a3fb23eaa22f98b, entries=150, sequenceid=586, filesize=11.9 K 2023-04-20 20:56:27,909 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 819071c32df35da7c00fe2675f028bf5 in 492ms, sequenceid=586, compaction requested=true 2023-04-20 20:56:27,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:27,909 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:27,916 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,916 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:27,916 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,916 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c157894[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:27,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:27,918 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,919 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:27,919 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,919 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:27,921 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:27,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:27,924 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:27,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:27,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:27,924 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:27,924 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:27,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,925 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:27,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:27,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:27,926 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,927 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:27,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:27,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:27,930 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:27,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:27,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,931 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:27,931 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:27,930 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:27,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:27,931 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:27,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:28,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-20 20:56:28,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024187692, max timestamp=1682024188003Num uniques 1077; 2023-04-20 20:56:28,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024187692, max timestamp=1682024188003Num uniques 1077; 2023-04-20 20:56:28,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-20 20:56:28,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:28,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:28,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:28,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:28,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:28,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-20 20:56:28,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682024187692, max timestamp=1682024188003Num uniques 1077; 2023-04-20 20:56:28,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-20 20:56:28,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:28,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024188013, max timestamp=1682024188027Num uniques 859; 2023-04-20 20:56:28,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:28,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024188013, max timestamp=1682024188027Num uniques 859; 2023-04-20 20:56:28,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682024188013, max timestamp=1682024188027Num uniques 859; 2023-04-20 20:56:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:28,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 20:56:28,031 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:28,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:28,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:28,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:28,034 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:28,034 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,034 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:28,034 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:28,035 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,035 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,036 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,037 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:28,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:28,038 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,038 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:28,038 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:28,038 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:28,040 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@793c1cb[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:28,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:28,041 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,046 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:28,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:28,046 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,046 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:28,046 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:28,047 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:28,047 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,047 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,047 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:28,050 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:28,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:28,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:28,050 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:28,050 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:28,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,051 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,052 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,053 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,054 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:28,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:28,055 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,055 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,055 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:28,055 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682024188063, max timestamp=1682024188070Num uniques 1118; 2023-04-20 20:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-20 20:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682024188063, max timestamp=1682024188070Num uniques 1118; 2023-04-20 20:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682024188063, max timestamp=1682024188070Num uniques 1118; 2023-04-20 20:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-20 20:56:28,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-20 20:56:28,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024188081, max timestamp=1682024188088Num uniques 1127; 2023-04-20 20:56:28,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-20 20:56:28,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024188081, max timestamp=1682024188088Num uniques 1127; 2023-04-20 20:56:28,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-20 20:56:28,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682024188081, max timestamp=1682024188088Num uniques 1127; 2023-04-20 20:56:28,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-20 20:56:28,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024188100, max timestamp=1682024188106Num uniques 1086; 2023-04-20 20:56:28,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:56:28,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024188100, max timestamp=1682024188106Num uniques 1086; 2023-04-20 20:56:28,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:56:28,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682024188100, max timestamp=1682024188106Num uniques 1086; 2023-04-20 20:56:28,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 20:56:28,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682024188115, max timestamp=1682024188122Num uniques 1095; 2023-04-20 20:56:28,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-20 20:56:28,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682024188115, max timestamp=1682024188122Num uniques 1095; 2023-04-20 20:56:28,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682024188115, max timestamp=1682024188122Num uniques 1095; 2023-04-20 20:56:28,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-20 20:56:28,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-20 20:56:28,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024188135, max timestamp=1682024188141Num uniques 1154; 2023-04-20 20:56:28,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-20 20:56:28,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024188135, max timestamp=1682024188141Num uniques 1154; 2023-04-20 20:56:28,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-20 20:56:28,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682024188135, max timestamp=1682024188141Num uniques 1154; 2023-04-20 20:56:28,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-20 20:56:28,156 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:28,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:28,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:28,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:28,159 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:28,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:28,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:28,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,159 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,161 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682024188147, max timestamp=1682024188160Num uniques 1163; 2023-04-20 20:56:28,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682024188147, max timestamp=1682024188160Num uniques 1163; 2023-04-20 20:56:28,163 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=14328, suffix heap size=148920,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=313 2023-04-20 20:56:28,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=14328, suffix heap size=148920,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=313 2023-04-20 20:56:28,163 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:28,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682024188147, max timestamp=1682024188160Num uniques 1163; 2023-04-20 20:56:28,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=7458, suffix heap size=148920,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=163 2023-04-20 20:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:28,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,166 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:28,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:28,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:28,180 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,180 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:28,180 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:28,180 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:28,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:28,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:28,180 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:28,182 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1682024188165, max timestamp=1682024188182Num uniques 1172; 2023-04-20 20:56:28,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-20 20:56:28,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682024188165, max timestamp=1682024188182Num uniques 1172; 2023-04-20 20:56:28,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-04-20 20:56:28,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024188165, max timestamp=1682024188182Num uniques 1022; 2023-04-20 20:56:28,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=6870, suffix heap size=134256,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=150 2023-04-20 20:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:28,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f57796f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:28,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:28,190 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,191 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:28,191 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:28,191 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,191 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:28,192 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,192 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:28,194 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:28,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:28,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:28,195 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:28,195 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:28,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,197 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,198 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,199 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:28,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:28,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,201 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,201 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:28,202 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:28,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,201 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,202 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:28,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1682024188195, max timestamp=1682024188203Num uniques 1131; 2023-04-20 20:56:28,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=15156, suffix heap size=145592,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=331 2023-04-20 20:56:28,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1682024188195, max timestamp=1682024188203Num uniques 1081; 2023-04-20 20:56:28,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1682024188195, max timestamp=1682024188203Num uniques 1009; 2023-04-20 20:56:28,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-20 20:56:28,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15156, suffix heap size=132904,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=331 2023-04-20 20:56:28,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024248218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682024188214, max timestamp=1682024188222Num uniques 1190; 2023-04-20 20:56:28,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-20 20:56:28,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682024188217, max timestamp=1682024188222Num uniques 1190; 2023-04-20 20:56:28,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=8700, suffix heap size=151728,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=190 2023-04-20 20:56:28,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682024188217, max timestamp=1682024188222Num uniques 1190; 2023-04-20 20:56:28,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=8700, suffix heap size=151728,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=190 2023-04-20 20:56:28,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024188236, max timestamp=1682024188242Num uniques 1149; 2023-04-20 20:56:28,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024188236, max timestamp=1682024188242Num uniques 1049; 2023-04-20 20:56:28,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-20 20:56:28,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682024188236, max timestamp=1682024188242Num uniques 1049; 2023-04-20 20:56:28,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:56:28,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-20 20:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682024188262, max timestamp=1682024188269Num uniques 1108; 2023-04-20 20:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682024188262, max timestamp=1682024188269Num uniques 1108; 2023-04-20 20:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682024188262, max timestamp=1682024188269Num uniques 1108; 2023-04-20 20:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-20 20:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-20 20:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-20 20:56:28,302 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:28,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024188289, max timestamp=1682024188300Num uniques 1067; 2023-04-20 20:56:28,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-20 20:56:28,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024188289, max timestamp=1682024188300Num uniques 1067; 2023-04-20 20:56:28,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682024188289, max timestamp=1682024188300Num uniques 1067; 2023-04-20 20:56:28,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-20 20:56:28,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-20 20:56:28,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:28,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:28,308 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:28,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:28,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:28,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,309 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:28,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:28,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:28,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:28,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:28,319 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,321 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4734333a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:28,324 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:28,324 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,327 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:28,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,327 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:28,329 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024248325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682024188322, max timestamp=1682024188330Num uniques 1076; 2023-04-20 20:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682024188322, max timestamp=1682024188330Num uniques 1076; 2023-04-20 20:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682024188322, max timestamp=1682024188330Num uniques 1076; 2023-04-20 20:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-20 20:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-20 20:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-20 20:56:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:28,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:28,334 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:28,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:28,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:28,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:28,335 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:28,337 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,337 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,337 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,338 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,339 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,341 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:28,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,341 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,342 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:28,342 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:28,342 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:28,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024188344, max timestamp=1682024188358Num uniques 1085; 2023-04-20 20:56:28,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024188344, max timestamp=1682024188358Num uniques 1085; 2023-04-20 20:56:28,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-20 20:56:28,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682024188344, max timestamp=1682024188358Num uniques 1085; 2023-04-20 20:56:28,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-20 20:56:28,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-20 20:56:28,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1682024188377, max timestamp=1682024188386Num uniques 1094; 2023-04-20 20:56:28,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-20 20:56:28,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1682024188377, max timestamp=1682024188386Num uniques 1094; 2023-04-20 20:56:28,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-20 20:56:28,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1682024188377, max timestamp=1682024188386Num uniques 1094; 2023-04-20 20:56:28,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-20 20:56:28,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024248406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682024188399, max timestamp=1682024188409Num uniques 1153; 2023-04-20 20:56:28,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-20 20:56:28,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682024188399, max timestamp=1682024188409Num uniques 1153; 2023-04-20 20:56:28,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-20 20:56:28,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1682024188399, max timestamp=1682024188409Num uniques 1153; 2023-04-20 20:56:28,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-04-20 20:56:28,429 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420b1e8074a57484d059c376aa11edf5fe0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b1e8074a57484d059c376aa11edf5fe0 2023-04-20 20:56:28,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1682024188414, max timestamp=1682024188438Num uniques 1112; 2023-04-20 20:56:28,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1682024188414, max timestamp=1682024188438Num uniques 1112; 2023-04-20 20:56:28,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=14282, suffix heap size=143616,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=312 2023-04-20 20:56:28,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=14282, suffix heap size=143616,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=312 2023-04-20 20:56:28,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1682024188414, max timestamp=1682024188438Num uniques 1112; 2023-04-20 20:56:28,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=14282, suffix heap size=143616,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=312 2023-04-20 20:56:28,442 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:28,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:28,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:28,446 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:28,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:28,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:28,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,446 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,448 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,448 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,449 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:28,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:28,450 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,451 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:28,451 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:28,451 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:28,452 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,452 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,453 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40623290[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:28,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:28,454 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:28,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,456 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:28,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1682024188455, max timestamp=1682024188463Num uniques 1171; 2023-04-20 20:56:28,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=14696, suffix heap size=149752,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=321 2023-04-20 20:56:28,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1682024188455, max timestamp=1682024188463Num uniques 1171; 2023-04-20 20:56:28,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=14696, suffix heap size=149752,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=321 2023-04-20 20:56:28,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1682024188455, max timestamp=1682024188463Num uniques 1171; 2023-04-20 20:56:28,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=14696, suffix heap size=149752,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=321 2023-04-20 20:56:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:28,467 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:28,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:28,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:28,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:28,467 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:28,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,469 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,469 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,471 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,474 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:28,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:28,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,473 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,474 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:28,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,474 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:28,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024248489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024248490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024248490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682024188487, max timestamp=1682024188495Num uniques 1180; 2023-04-20 20:56:28,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=8240, suffix heap size=150688,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=180 2023-04-20 20:56:28,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682024188487, max timestamp=1682024188495Num uniques 1180; 2023-04-20 20:56:28,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682024188487, max timestamp=1682024188495Num uniques 1180; 2023-04-20 20:56:28,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=8240, suffix heap size=150688,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=180 2023-04-20 20:56:28,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=8240, suffix heap size=150688,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=180 2023-04-20 20:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682024188518, max timestamp=1682024188549Num uniques 1039; 2023-04-20 20:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-20 20:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682024188518, max timestamp=1682024188549Num uniques 1039; 2023-04-20 20:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-20 20:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682024188518, max timestamp=1682024188549Num uniques 1039; 2023-04-20 20:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-20 20:56:28,575 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:28,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:28,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:28,580 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:28,580 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,580 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:28,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:28,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,581 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,582 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,582 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,583 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,583 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:28,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:28,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:28,585 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,585 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:28,585 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:28,585 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:28,588 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,588 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,589 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,589 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e62ece[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:28,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:28,590 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682024188575, max timestamp=1682024188589Num uniques 1098; 2023-04-20 20:56:28,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682024188575, max timestamp=1682024188589Num uniques 1098; 2023-04-20 20:56:28,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-20 20:56:28,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682024188575, max timestamp=1682024188589Num uniques 1098; 2023-04-20 20:56:28,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-20 20:56:28,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-20 20:56:28,593 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,593 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:28,593 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:28,593 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:28,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,593 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:28,594 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:28,597 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:28,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:28,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:28,597 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:28,597 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:28,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,599 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,600 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,601 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,604 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:28,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,604 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:28,604 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,604 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682024188617, max timestamp=1682024188622Num uniques 1107; 2023-04-20 20:56:28,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-20 20:56:28,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682024188617, max timestamp=1682024188622Num uniques 1107; 2023-04-20 20:56:28,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682024188617, max timestamp=1682024188622Num uniques 1107; 2023-04-20 20:56:28,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-20 20:56:28,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-20 20:56:28,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1682024188645, max timestamp=1682024188655Num uniques 1066; 2023-04-20 20:56:28,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=14466, suffix heap size=138832,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=316 2023-04-20 20:56:28,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682024188645, max timestamp=1682024188655Num uniques 1066; 2023-04-20 20:56:28,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-20 20:56:28,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1682024188645, max timestamp=1682024188655Num uniques 1066; 2023-04-20 20:56:28,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=14466, suffix heap size=138832,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=316 2023-04-20 20:56:28,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1682024188670, max timestamp=1682024188680Num uniques 1175; 2023-04-20 20:56:28,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-04-20 20:56:28,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1682024188670, max timestamp=1682024188680Num uniques 1175; 2023-04-20 20:56:28,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-04-20 20:56:28,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1682024188675, max timestamp=1682024188680Num uniques 1075; 2023-04-20 20:56:28,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-04-20 20:56:28,704 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:28,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:28,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1682024188698, max timestamp=1682024188705Num uniques 1134; 2023-04-20 20:56:28,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=13004, suffix heap size=145904,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=284 2023-04-20 20:56:28,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1682024188699, max timestamp=1682024188705Num uniques 1134; 2023-04-20 20:56:28,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=8424, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-04-20 20:56:28,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024188699, max timestamp=1682024188705Num uniques 1034; 2023-04-20 20:56:28,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=6870, suffix heap size=135504,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=150 2023-04-20 20:56:28,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:28,711 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:28,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:28,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:28,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,712 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,714 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:28,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:28,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:28,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:28,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:28,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:28,720 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:28,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,721 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:28,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4deea89c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:28,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:28,724 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,725 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:28,725 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,725 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:28,732 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:28,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,732 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1682024188724, max timestamp=1682024188734Num uniques 1043; 2023-04-20 20:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=11118, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-20 20:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1682024188724, max timestamp=1682024188734Num uniques 1143; 2023-04-20 20:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52338, new segment data size=11118, suffix heap size=146840,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1143, new segment cells count=243 2023-04-20 20:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1682024188724, max timestamp=1682024188734Num uniques 1009; 2023-04-20 20:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=11118, suffix heap size=132904,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=243 2023-04-20 20:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:28,739 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:28,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:28,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:28,739 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:28,739 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:28,741 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,741 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,742 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,743 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,746 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:28,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:28,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,751 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,752 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:28,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,752 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,752 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1682024188749, max timestamp=1682024188765Num uniques 1102; 2023-04-20 20:56:28,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=13830, suffix heap size=142576,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=302 2023-04-20 20:56:28,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1682024188749, max timestamp=1682024188765Num uniques 1102; 2023-04-20 20:56:28,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=13830, suffix heap size=142576,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=302 2023-04-20 20:56:28,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1682024188753, max timestamp=1682024188765Num uniques 1102; 2023-04-20 20:56:28,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-20 20:56:28,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1682024188781, max timestamp=1682024188786Num uniques 1161; 2023-04-20 20:56:28,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=11946, suffix heap size=148712,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=261 2023-04-20 20:56:28,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1682024188781, max timestamp=1682024188786Num uniques 1011; 2023-04-20 20:56:28,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=11946, suffix heap size=133112,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=261 2023-04-20 20:56:28,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1682024188781, max timestamp=1682024188786Num uniques 1161; 2023-04-20 20:56:28,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=11946, suffix heap size=148712,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=261 2023-04-20 20:56:28,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1682024188807, max timestamp=1682024188819Num uniques 1120; 2023-04-20 20:56:28,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1682024188807, max timestamp=1682024188815Num uniques 1120; 2023-04-20 20:56:28,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=7780, suffix heap size=144448,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=170 2023-04-20 20:56:28,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=7780, suffix heap size=144448,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=170 2023-04-20 20:56:28,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024188807, max timestamp=1682024188815Num uniques 1120; 2023-04-20 20:56:28,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=6870, suffix heap size=144448,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=150 2023-04-20 20:56:28,834 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=641, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/80e5c4903d3b4270b3a292c6e20014c7 2023-04-20 20:56:28,852 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:28,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682024188838, max timestamp=1682024188850Num uniques 1029; 2023-04-20 20:56:28,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682024188838, max timestamp=1682024188850Num uniques 1029; 2023-04-20 20:56:28,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-04-20 20:56:28,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-04-20 20:56:28,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682024188838, max timestamp=1682024188850Num uniques 1009; 2023-04-20 20:56:28,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10484, suffix heap size=132904,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=229 2023-04-20 20:56:28,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:28,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:28,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:28,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:28,860 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:28,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:28,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:28,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,860 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,863 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:28,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:28,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:28,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/dae0a27ea8794bc5b9569699e78585c9 2023-04-20 20:56:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:28,872 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,872 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:28,872 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:28,872 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:28,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:28,875 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:28,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba5eff8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:28,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:28,878 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=338, cellSize=15478, totalHeapSize=35392, min timestamp=1682024188867, max timestamp=1682024188878Num uniques 1088; 2023-04-20 20:56:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=15478, suffix heap size=141120,new segment heap size=35392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=338 2023-04-20 20:56:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=338, cellSize=15478, totalHeapSize=35392, min timestamp=1682024188867, max timestamp=1682024188878Num uniques 1088; 2023-04-20 20:56:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=15478, suffix heap size=141120,new segment heap size=35392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=338 2023-04-20 20:56:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=338, cellSize=15478, totalHeapSize=35392, min timestamp=1682024188867, max timestamp=1682024188878Num uniques 1088; 2023-04-20 20:56:28,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=15478, suffix heap size=141120,new segment heap size=35392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=338 2023-04-20 20:56:28,882 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:28,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:28,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:28,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:28,884 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:28,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:28,888 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:28,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:28,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:28,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:28,888 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:28,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,890 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:28,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,891 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,893 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,900 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:28,900 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:28,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:28,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:28,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:28,901 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:28,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:28,901 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:28,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:28,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:28,901 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:28,901 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:28,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024248906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:28,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024248907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:28,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9002, totalHeapSize=20728, min timestamp=1682024188901, max timestamp=1682024188911Num uniques 1197; 2023-04-20 20:56:28,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54802, new segment data size=9002, suffix heap size=152456,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1197, new segment cells count=197 2023-04-20 20:56:28,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9004, totalHeapSize=20728, min timestamp=1682024188901, max timestamp=1682024188911Num uniques 1197; 2023-04-20 20:56:28,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9002, totalHeapSize=20728, min timestamp=1682024188901, max timestamp=1682024188911Num uniques 1197; 2023-04-20 20:56:28,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54804, new segment data size=9004, suffix heap size=152456,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1197, new segment cells count=197 2023-04-20 20:56:28,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54802, new segment data size=9002, suffix heap size=152456,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1197, new segment cells count=197 2023-04-20 20:56:28,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1682024188944, max timestamp=1682024188965Num uniques 1056; 2023-04-20 20:56:28,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1682024188944, max timestamp=1682024188965Num uniques 1056; 2023-04-20 20:56:28,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-04-20 20:56:28,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1682024188944, max timestamp=1682024188965Num uniques 1056; 2023-04-20 20:56:28,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-04-20 20:56:28,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-04-20 20:56:28,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1682024188975, max timestamp=1682024188996Num uniques 1015; 2023-04-20 20:56:28,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=9840, suffix heap size=133528,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=215 2023-04-20 20:56:28,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1682024188975, max timestamp=1682024188996Num uniques 1015; 2023-04-20 20:56:28,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=9840, suffix heap size=133528,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=215 2023-04-20 20:56:28,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682024188975, max timestamp=1682024188996Num uniques 1015; 2023-04-20 20:56:28,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=6870, suffix heap size=133528,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=150 2023-04-20 20:56:29,001 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:29,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:29,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:29,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:29,005 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:29,005 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,005 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:29,005 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:29,005 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,005 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,006 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,006 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,007 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:29,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:29,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:29,008 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,008 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:29,009 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:29,009 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:29,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:29,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:29,009 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:29,012 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:29,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a82beae[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:29,014 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:29,014 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,015 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:29,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,018 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:29,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:29,020 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:29,020 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:29,020 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:29,020 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:29,020 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:29,022 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,022 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,023 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,027 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,027 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,027 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:29,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:29,027 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,027 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,028 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:29,028 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:29,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:29,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7964, totalHeapSize=18336, min timestamp=1682024189017, max timestamp=1682024189029Num uniques 1074; 2023-04-20 20:56:29,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=7964, suffix heap size=139664,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=174 2023-04-20 20:56:29,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7964, totalHeapSize=18336, min timestamp=1682024189017, max timestamp=1682024189029Num uniques 1074; 2023-04-20 20:56:29,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=7964, suffix heap size=139664,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=174 2023-04-20 20:56:29,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7964, totalHeapSize=18336, min timestamp=1682024189017, max timestamp=1682024189029Num uniques 1009; 2023-04-20 20:56:29,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7964, suffix heap size=132904,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=174 2023-04-20 20:56:29,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1682024189065, max timestamp=1682024189073Num uniques 1033; 2023-04-20 20:56:29,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47308, new segment data size=10668, suffix heap size=135400,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=233 2023-04-20 20:56:29,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1682024189065, max timestamp=1682024189073Num uniques 1033; 2023-04-20 20:56:29,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47308, new segment data size=10668, suffix heap size=135400,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=233 2023-04-20 20:56:29,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1682024189065, max timestamp=1682024189073Num uniques 1033; 2023-04-20 20:56:29,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47308, new segment data size=10668, suffix heap size=135400,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=233 2023-04-20 20:56:29,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1682024189085, max timestamp=1682024189099Num uniques 1092; 2023-04-20 20:56:29,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1682024189085, max timestamp=1682024189099Num uniques 1092; 2023-04-20 20:56:29,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=8792, suffix heap size=141536,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=192 2023-04-20 20:56:29,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=8792, suffix heap size=141536,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=192 2023-04-20 20:56:29,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1682024189085, max timestamp=1682024189099Num uniques 1092; 2023-04-20 20:56:29,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=8792, suffix heap size=141536,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=192 2023-04-20 20:56:29,128 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1682024189115, max timestamp=1682024189125Num uniques 1051; 2023-04-20 20:56:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1682024189115, max timestamp=1682024189125Num uniques 1051; 2023-04-20 20:56:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1682024189115, max timestamp=1682024189125Num uniques 1051; 2023-04-20 20:56:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=11495, suffix heap size=137272,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=251 2023-04-20 20:56:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=11495, suffix heap size=137272,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=251 2023-04-20 20:56:29,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:29,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=11495, suffix heap size=137272,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=251 2023-04-20 20:56:29,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:29,132 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:29,132 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,132 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:29,132 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:29,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,133 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,134 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,134 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,134 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,134 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:29,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:29,136 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,137 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:29,137 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:29,137 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:29,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:29,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:29,137 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:29,140 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,140 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,141 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:29,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12fa7f6e[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:29,142 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:29,142 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,143 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:29,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:29,145 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,146 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:29,146 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:29,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:29,149 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:29,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:29,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:29,149 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:29,149 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:29,151 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,151 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,151 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,153 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1682024189138, max timestamp=1682024189152Num uniques 1110; 2023-04-20 20:56:29,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=11900, suffix heap size=143408,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=260 2023-04-20 20:56:29,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1682024189138, max timestamp=1682024189152Num uniques 1110; 2023-04-20 20:56:29,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1682024189138, max timestamp=1682024189152Num uniques 1110; 2023-04-20 20:56:29,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=11900, suffix heap size=143408,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=260 2023-04-20 20:56:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=11900, suffix heap size=143408,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=260 2023-04-20 20:56:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,157 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,160 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,161 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:29,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,161 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,161 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:29,162 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:29,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=319, cellSize=14604, totalHeapSize=33416, min timestamp=1682024189159, max timestamp=1682024189180Num uniques 1119; 2023-04-20 20:56:29,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51244, new segment data size=14604, suffix heap size=144344,new segment heap size=33416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1119, new segment cells count=319 2023-04-20 20:56:29,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=319, cellSize=14604, totalHeapSize=33416, min timestamp=1682024189159, max timestamp=1682024189180Num uniques 1119; 2023-04-20 20:56:29,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=319, cellSize=14604, totalHeapSize=33416, min timestamp=1682024189159, max timestamp=1682024189180Num uniques 1119; 2023-04-20 20:56:29,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51244, new segment data size=14604, suffix heap size=144344,new segment heap size=33416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1119, new segment cells count=319 2023-04-20 20:56:29,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51244, new segment data size=14604, suffix heap size=144344,new segment heap size=33416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1119, new segment cells count=319 2023-04-20 20:56:29,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=278, cellSize=12728, totalHeapSize=29152, min timestamp=1682024189191, max timestamp=1682024189208Num uniques 1178; 2023-04-20 20:56:29,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53948, new segment data size=12728, suffix heap size=150480,new segment heap size=29152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1178, new segment cells count=278 2023-04-20 20:56:29,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=278, cellSize=12728, totalHeapSize=29152, min timestamp=1682024189191, max timestamp=1682024189208Num uniques 1178; 2023-04-20 20:56:29,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53948, new segment data size=12728, suffix heap size=150480,new segment heap size=29152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1178, new segment cells count=278 2023-04-20 20:56:29,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=278, cellSize=12728, totalHeapSize=29152, min timestamp=1682024189191, max timestamp=1682024189208Num uniques 1178; 2023-04-20 20:56:29,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53948, new segment data size=12728, suffix heap size=150480,new segment heap size=29152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1178, new segment cells count=278 2023-04-20 20:56:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=337, cellSize=15432, totalHeapSize=35288, min timestamp=1682024189225, max timestamp=1682024189236Num uniques 1137; 2023-04-20 20:56:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=337, cellSize=15432, totalHeapSize=35288, min timestamp=1682024189225, max timestamp=1682024189236Num uniques 1137; 2023-04-20 20:56:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52072, new segment data size=15432, suffix heap size=146216,new segment heap size=35288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1137, new segment cells count=337 2023-04-20 20:56:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=337, cellSize=15432, totalHeapSize=35288, min timestamp=1682024189225, max timestamp=1682024189236Num uniques 1137; 2023-04-20 20:56:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52072, new segment data size=15432, suffix heap size=146216,new segment heap size=35288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1137, new segment cells count=337 2023-04-20 20:56:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52072, new segment data size=15432, suffix heap size=146216,new segment heap size=35288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1137, new segment cells count=337 2023-04-20 20:56:29,262 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:29,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:29,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024249258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024249259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024249260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1682024189245, max timestamp=1682024189263Num uniques 1196; 2023-04-20 20:56:29,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54776, new segment data size=11266, suffix heap size=152352,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1196, new segment cells count=246 2023-04-20 20:56:29,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:29,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1682024189245, max timestamp=1682024189263Num uniques 1196; 2023-04-20 20:56:29,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54776, new segment data size=11266, suffix heap size=152352,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1196, new segment cells count=246 2023-04-20 20:56:29,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=196, cellSize=8976, totalHeapSize=20624, min timestamp=1682024189245, max timestamp=1682024189263Num uniques 1196; 2023-04-20 20:56:29,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54776, new segment data size=8976, suffix heap size=152352,new segment heap size=20624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1196, new segment cells count=196 2023-04-20 20:56:29,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:29,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:29,269 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:29,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:29,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:29,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,269 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,271 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,271 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,272 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:29,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:29,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:29,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:29,293 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,293 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:29,293 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:29,294 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:29,295 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,295 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:29,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a688fe5[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:29,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:29,298 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:29,302 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:29,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:29,303 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:29,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:29,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:29,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:29,303 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:29,304 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,305 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,305 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,306 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,313 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,314 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,314 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:29,313 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:29,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:29,314 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:29,314 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:29,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/b9948ad2555d4455923bac0f69c45ca0 2023-04-20 20:56:29,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/80e5c4903d3b4270b3a292c6e20014c7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/80e5c4903d3b4270b3a292c6e20014c7 2023-04-20 20:56:29,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=205, cellSize=9385, totalHeapSize=21560, min timestamp=1682024189299, max timestamp=1682024189321Num uniques 1105; 2023-04-20 20:56:29,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=9385, suffix heap size=142888,new segment heap size=21560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=205 2023-04-20 20:56:29,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=205, cellSize=9385, totalHeapSize=21560, min timestamp=1682024189299, max timestamp=1682024189321Num uniques 1105; 2023-04-20 20:56:29,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=9385, suffix heap size=142888,new segment heap size=21560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=205 2023-04-20 20:56:29,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1682024189299, max timestamp=1682024189321Num uniques 1055; 2023-04-20 20:56:29,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48315, new segment data size=7095, suffix heap size=137688,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1055, new segment cells count=155 2023-04-20 20:56:29,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/80e5c4903d3b4270b3a292c6e20014c7, entries=150, sequenceid=641, filesize=25.4 K 2023-04-20 20:56:29,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/dae0a27ea8794bc5b9569699e78585c9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/dae0a27ea8794bc5b9569699e78585c9 2023-04-20 20:56:29,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/dae0a27ea8794bc5b9569699e78585c9, entries=150, sequenceid=641, filesize=11.9 K 2023-04-20 20:56:29,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/b9948ad2555d4455923bac0f69c45ca0 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b9948ad2555d4455923bac0f69c45ca0 2023-04-20 20:56:29,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b9948ad2555d4455923bac0f69c45ca0, entries=150, sequenceid=641, filesize=11.9 K 2023-04-20 20:56:29,357 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~64.50 KB/66048, currentSize=78.27 KB/80150 for 819071c32df35da7c00fe2675f028bf5 in 1354ms, sequenceid=641, compaction requested=true 2023-04-20 20:56:29,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:29,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:29,358 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:56:29,358 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:29,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:29,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:29,359 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:29,359 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:29,359 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,359 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/830bc03e57244fe8bc3cb2ef162240b5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/4eb2780f57c845bba935398d8eeae271, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3d6ad2996c5742e8b524370df9323bbe, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/dae0a27ea8794bc5b9569699e78585c9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=48.1 K 2023-04-20 20:56:29,361 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 830bc03e57244fe8bc3cb2ef162240b5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682024185840 2023-04-20 20:56:29,362 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 4eb2780f57c845bba935398d8eeae271, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682024185911 2023-04-20 20:56:29,364 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156652 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:56:29,365 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/A is initiating minor compaction (all files) 2023-04-20 20:56:29,365 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/A in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,365 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0cfeeb95d9634a4b993c0384568cfbe6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/7c1c2e7363a045e4b267c30323749141, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6037f07d1d9e46fb8ee8a5ce9b870076, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/1683d40670104f5196fbf25657da4e3c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6fb5ba04dd34496b9798d6cebea15d62, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/80e5c4903d3b4270b3a292c6e20014c7] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=153.0 K 2023-04-20 20:56:29,365 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3d6ad2996c5742e8b524370df9323bbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682024186538 2023-04-20 20:56:29,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=84.98 KB heapSize=236.14 KB 2023-04-20 20:56:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:29,366 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:29,366 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:29,366 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:29,366 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0cfeeb95d9634a4b993c0384568cfbe6, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682024184655 2023-04-20 20:56:29,366 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:29,366 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:29,366 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:29,366 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting dae0a27ea8794bc5b9569699e78585c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682024187694 2023-04-20 20:56:29,367 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 7c1c2e7363a045e4b267c30323749141, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1682024184710 2023-04-20 20:56:29,368 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6037f07d1d9e46fb8ee8a5ce9b870076, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682024185840 2023-04-20 20:56:29,368 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 1683d40670104f5196fbf25657da4e3c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682024185911 2023-04-20 20:56:29,369 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 6fb5ba04dd34496b9798d6cebea15d62, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682024186538 2023-04-20 20:56:29,370 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 80e5c4903d3b4270b3a292c6e20014c7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682024187694 2023-04-20 20:56:29,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024249388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024249392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,401 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#593 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:29,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024249394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024249396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024249396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,414 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:29,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:29,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:29,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:29,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:29,424 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:29,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:29,424 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:29,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,425 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,427 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:29,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:29,449 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#A#compaction#592 average throughput is 0.35 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:29,451 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/99a44a6830b640299a5fff4b79e93acb as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/99a44a6830b640299a5fff4b79e93acb 2023-04-20 20:56:29,456 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into 99a44a6830b640299a5fff4b79e93acb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:29,456 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:29,456 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=12, startTime=1682024189358; duration=0sec 2023-04-20 20:56:29,456 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:29,456 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:29,457 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:29,458 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/C is initiating minor compaction (all files) 2023-04-20 20:56:29,458 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/C in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,458 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/adb476adccf0456b8e8c0ac33a070fa5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/95edeb826ece4ce58b4e6275744e6416, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9e96caffcccd49cc8a3fb23eaa22f98b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b9948ad2555d4455923bac0f69c45ca0] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=48.1 K 2023-04-20 20:56:29,458 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting adb476adccf0456b8e8c0ac33a070fa5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682024185840 2023-04-20 20:56:29,458 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 95edeb826ece4ce58b4e6275744e6416, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1682024185911 2023-04-20 20:56:29,459 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9e96caffcccd49cc8a3fb23eaa22f98b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682024186538 2023-04-20 20:56:29,459 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b9948ad2555d4455923bac0f69c45ca0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682024187694 2023-04-20 20:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:29,468 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,468 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:29,468 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:29,468 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,470 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:29,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,472 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#C#compaction#595 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e366f7f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:29,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:29,473 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,474 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:29,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,475 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:29,476 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:29,479 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:29,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:29,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:29,479 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:29,479 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:29,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,482 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,483 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,485 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,488 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:29,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,488 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:29,488 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,488 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,489 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024249497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024249500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024249503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024249506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024249507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,537 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/cc27dabe4434407ead1de06183cca8a8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/cc27dabe4434407ead1de06183cca8a8 2023-04-20 20:56:29,542 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/C of 819071c32df35da7c00fe2675f028bf5 into cc27dabe4434407ead1de06183cca8a8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:29,542 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:29,543 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/C, priority=12, startTime=1682024189358; duration=0sec 2023-04-20 20:56:29,543 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:29,558 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/0b6bce97acf84c0a9d24708b59795782 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0b6bce97acf84c0a9d24708b59795782 2023-04-20 20:56:29,563 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/A of 819071c32df35da7c00fe2675f028bf5 into 0b6bce97acf84c0a9d24708b59795782(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:29,563 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:29,563 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/A, priority=10, startTime=1682024189358; duration=0sec 2023-04-20 20:56:29,563 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:29,588 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:29,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:29,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:29,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:29,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:29,592 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:29,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:29,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:29,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,594 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,594 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:29,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:29,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:29,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:29,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:29,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,600 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:29,612 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,612 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:29,612 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:29,612 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:29,614 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,616 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412ed3d9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:29,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:29,617 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,621 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,621 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:29,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:29,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:29,621 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:29,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:29,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,623 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:29,623 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:29,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:29,626 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:29,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:29,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:29,626 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:29,626 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:29,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,628 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,630 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,632 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,635 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,635 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,635 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,635 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:29,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,635 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:29,636 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,636 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,636 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,636 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,636 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,636 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:29,636 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:29,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024249701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024249707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024249709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024249710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:29,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024249712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,736 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:29,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:29,741 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:29,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:29,741 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:29,742 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,742 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,744 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,744 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,745 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:29,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:29,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:29,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:29,749 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ed59d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:29,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:29,751 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,753 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:29,753 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,754 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:29,755 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:29,758 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:29,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:29,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:29,758 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:29,758 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:29,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,761 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,762 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,765 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,769 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,770 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,770 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:29,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:29,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:29,771 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:29,771 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:29,869 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420ba78a63224ca4105a99673ea47110ab9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ba78a63224ca4105a99673ea47110ab9 2023-04-20 20:56:29,871 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:29,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:29,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:29,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:29,876 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:29,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:29,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:29,876 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1427, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/2b883f3a48ab4f12bca74f87ba8250f7 2023-04-20 20:56:29,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,877 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,881 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,882 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:29,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:29,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:29,884 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:29,890 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:29,893 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,894 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,894 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9765d2[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:29,896 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:29,896 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,897 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,897 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:29,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,898 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:29,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:29,899 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,899 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,900 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:29,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:29,902 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:29,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:29,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:29,902 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:29,902 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:29,904 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,904 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:29,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,904 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:29,905 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,907 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:29,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:29,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:29,910 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:29,910 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:29,910 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:29,910 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024250005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,010 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:30,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:30,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024250010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024250012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:30,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:30,014 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:30,014 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,014 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:30,014 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:30,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,015 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,016 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,016 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,017 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,017 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:30,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024250016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:30,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:30,018 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,019 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:30,019 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:30,019 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:30,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024250017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,021 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7347a2e8[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:30,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:30,024 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,025 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:30,025 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,026 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:30,028 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:30,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,028 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:30,032 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:30,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:30,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:30,032 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:30,032 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:30,034 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,034 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,034 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,035 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,036 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,038 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,038 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,038 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:30,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:30,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:30,038 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,038 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,039 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:30,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,039 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:30,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,039 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,139 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:30,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:30,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:30,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:30,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:30,143 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:30,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:30,143 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:30,144 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,144 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,145 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,145 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,146 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:30,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:30,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:30,147 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,147 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:30,147 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:30,148 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:30,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:30,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:30,148 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:30,150 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,150 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,151 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,151 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c00742[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:30,153 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:30,153 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,154 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:30,154 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,154 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:30,156 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:30,160 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:30,160 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:30,160 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:30,160 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:30,160 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:30,162 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,162 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,162 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,163 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,164 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,169 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,169 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,170 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:30,170 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:30,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,270 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:30,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:30,274 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:30,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:30,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:30,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,274 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,276 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,276 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:30,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:30,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:30,278 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,278 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:30,278 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:30,278 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:30,282 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,283 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3274f07a[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:30,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:30,284 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,286 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:30,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:30,286 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:30,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,286 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,287 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:30,290 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:30,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:30,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:30,290 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:30,290 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:30,291 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,291 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,291 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,292 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1427 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/003ec7aea6f443659db9b8956f567866 2023-04-20 20:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,294 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,297 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:30,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:30,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,298 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:30,298 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:30,297 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,298 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:30,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1427 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/530c19bb0a614739a37a0fd461722697 2023-04-20 20:56:30,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/2b883f3a48ab4f12bca74f87ba8250f7 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2b883f3a48ab4f12bca74f87ba8250f7 2023-04-20 20:56:30,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2b883f3a48ab4f12bca74f87ba8250f7, entries=150, sequenceid=1427, filesize=25.4 K 2023-04-20 20:56:30,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/003ec7aea6f443659db9b8956f567866 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/003ec7aea6f443659db9b8956f567866 2023-04-20 20:56:30,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/003ec7aea6f443659db9b8956f567866, entries=150, sequenceid=1427, filesize=11.9 K 2023-04-20 20:56:30,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/530c19bb0a614739a37a0fd461722697 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/530c19bb0a614739a37a0fd461722697 2023-04-20 20:56:30,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/530c19bb0a614739a37a0fd461722697, entries=150, sequenceid=1427, filesize=11.9 K 2023-04-20 20:56:30,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~91.69 KB/93890, heapSize ~256.02 KB/262160, currentSize=93.93 KB/96180 for 819071c32df35da7c00fe2675f028bf5 in 957ms, sequenceid=1427, compaction requested=false 2023-04-20 20:56:30,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:30,398 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:30,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:30,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:30,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:30,401 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:30,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:30,401 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:30,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,402 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,404 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,404 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,405 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:30,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:30,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:30,406 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:30,406 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:30,407 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 20:56:30,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:30,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:30,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:30,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:30,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:30,407 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:30,421 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042045710b545616454096b7b647d89f5314 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042045710b545616454096b7b647d89f5314 2023-04-20 20:56:30,429 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1447, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/d20a7cd5588646b2a767a262a7e61b36 2023-04-20 20:56:30,441 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1447 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/3c48fded5cfc4ab59040f1c0c3213aa4 2023-04-20 20:56:30,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:30,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:30,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024250525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024250527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024250528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024250528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024250529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024250630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024250631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024250634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024250634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024250634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024250834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024250835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024250836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024250836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024250837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,854 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1447 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/9315636958ba4bffbf961bbe75f86db2 2023-04-20 20:56:30,858 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/d20a7cd5588646b2a767a262a7e61b36 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d20a7cd5588646b2a767a262a7e61b36 2023-04-20 20:56:30,861 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d20a7cd5588646b2a767a262a7e61b36, entries=150, sequenceid=1447, filesize=25.4 K 2023-04-20 20:56:30,862 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/3c48fded5cfc4ab59040f1c0c3213aa4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3c48fded5cfc4ab59040f1c0c3213aa4 2023-04-20 20:56:30,865 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3c48fded5cfc4ab59040f1c0c3213aa4, entries=150, sequenceid=1447, filesize=11.9 K 2023-04-20 20:56:30,866 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/9315636958ba4bffbf961bbe75f86db2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9315636958ba4bffbf961bbe75f86db2 2023-04-20 20:56:30,869 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9315636958ba4bffbf961bbe75f86db2, entries=150, sequenceid=1447, filesize=11.9 K 2023-04-20 20:56:30,870 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 819071c32df35da7c00fe2675f028bf5 in 463ms, sequenceid=1447, compaction requested=true 2023-04-20 20:56:30,870 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:30,870 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:30,872 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6798ba13[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:30,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:30,874 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,875 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:30,875 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:30,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,876 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:30,877 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:30,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:30,879 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:30,879 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:30,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:30,880 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:30,880 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:30,883 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,883 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,883 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:30,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,883 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:30,885 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:30,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:30,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:30,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:30,888 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:30,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:30,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:30,887 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:30,888 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,888 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:30,888 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:30,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:30,988 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:30,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:30,992 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:30,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:30,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:30,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,992 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,993 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,994 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:30,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:30,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:30,994 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:30,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:30,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:30,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:30,995 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:30,995 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:30,996 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 20:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:31,008 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420c5780cfc30d340d082efb9f30c436a66 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c5780cfc30d340d082efb9f30c436a66 2023-04-20 20:56:31,013 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1463, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/24d9438c0890458286dc421e3d150164 2023-04-20 20:56:31,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:31,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:31,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024251147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024251149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024251149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024251149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024251150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024251251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024251254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024251254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024251254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024251254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,426 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1463 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/b9285ed8adf744aead22e911eb60b615 2023-04-20 20:56:31,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024251454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024251455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024251456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024251457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024251458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024251758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024251758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024251760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024251761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:31,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024251761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,835 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1463 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/30d318a5665044b7a224e9f765f9aaa4 2023-04-20 20:56:31,839 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/24d9438c0890458286dc421e3d150164 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/24d9438c0890458286dc421e3d150164 2023-04-20 20:56:31,842 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/24d9438c0890458286dc421e3d150164, entries=150, sequenceid=1463, filesize=25.4 K 2023-04-20 20:56:31,843 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/b9285ed8adf744aead22e911eb60b615 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b9285ed8adf744aead22e911eb60b615 2023-04-20 20:56:31,847 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b9285ed8adf744aead22e911eb60b615, entries=150, sequenceid=1463, filesize=11.9 K 2023-04-20 20:56:31,847 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/30d318a5665044b7a224e9f765f9aaa4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/30d318a5665044b7a224e9f765f9aaa4 2023-04-20 20:56:31,850 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/30d318a5665044b7a224e9f765f9aaa4, entries=150, sequenceid=1463, filesize=11.9 K 2023-04-20 20:56:31,851 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 819071c32df35da7c00fe2675f028bf5 in 856ms, sequenceid=1463, compaction requested=true 2023-04-20 20:56:31,851 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:31,851 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:31,853 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:31,853 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298604ea[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:31,855 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:31,855 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:31,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,862 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:31,863 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:31,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:31,866 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:31,866 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:31,866 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:31,866 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:31,866 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:31,867 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,867 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:31,867 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:31,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:31,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:31,868 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:31,869 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:31,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:31,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:31,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:31,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:31,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:31,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:31,871 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:31,871 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:31,872 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:31,872 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:31,872 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,872 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:31,872 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:31,872 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,972 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:31,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:31,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:31,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:31,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:31,975 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:31,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:31,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:31,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:31,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:31,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:31,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:31,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:31,978 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:31,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:31,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:31,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:31,979 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:31,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:31,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:31,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:31,980 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:31,980 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:31,980 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:31,980 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 20:56:31,980 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:31,980 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:31,980 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:31,980 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:31,980 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:31,981 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:31,993 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420248aec14d26f462bae9c3788b30eb709 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420248aec14d26f462bae9c3788b30eb709 2023-04-20 20:56:32,005 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1479, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/fb2610d32e734357bc3b4d109d6474e8 2023-04-20 20:56:32,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:32,264 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:32,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024252275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024252277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024252279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024252279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024252280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024252381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024252381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024252382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024252382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024252382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,414 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1479 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/0f5cad6420e74d86b8cf7499e2e36819 2023-04-20 20:56:32,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024252585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024252585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024252585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024252585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024252585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,825 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1479 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/76a16bf719414227941e5b5a4b1eb35a 2023-04-20 20:56:32,829 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/fb2610d32e734357bc3b4d109d6474e8 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fb2610d32e734357bc3b4d109d6474e8 2023-04-20 20:56:32,833 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fb2610d32e734357bc3b4d109d6474e8, entries=150, sequenceid=1479, filesize=25.4 K 2023-04-20 20:56:32,834 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/0f5cad6420e74d86b8cf7499e2e36819 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/0f5cad6420e74d86b8cf7499e2e36819 2023-04-20 20:56:32,837 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/0f5cad6420e74d86b8cf7499e2e36819, entries=150, sequenceid=1479, filesize=11.9 K 2023-04-20 20:56:32,838 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/76a16bf719414227941e5b5a4b1eb35a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/76a16bf719414227941e5b5a4b1eb35a 2023-04-20 20:56:32,840 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/76a16bf719414227941e5b5a4b1eb35a, entries=150, sequenceid=1479, filesize=11.9 K 2023-04-20 20:56:32,841 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 819071c32df35da7c00fe2675f028bf5 in 861ms, sequenceid=1479, compaction requested=true 2023-04-20 20:56:32,841 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:32,841 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,843 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d38ec8f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:32,845 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:32,845 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,846 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:32,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:32,847 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:32,847 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,847 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:32,851 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:32,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:32,855 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:32,855 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:32,855 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:32,855 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:32,855 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:32,856 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:32,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,857 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:32,857 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,858 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:32,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:32,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:32,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:32,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:32,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:32,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:32,861 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:32,861 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:32,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:32,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:32,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:32,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:32,889 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 20:56:32,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:32,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:32,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:32,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:32,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:32,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:32,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024252898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024252898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024252901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024252902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:32,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024252902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,961 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:32,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:32,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:32,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:32,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:32,965 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:32,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:32,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:32,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:32,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,965 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,967 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:32,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:32,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:32,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:32,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:32,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:32,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:32,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:32,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:32,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:32,969 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:32,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:32,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:32,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:32,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:32,972 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:32,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:32,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785a3672[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:32,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:32,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:32,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:32,977 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:32,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:32,979 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:32,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:32,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:32,979 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:32,979 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:32,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,981 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:32,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:32,982 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:32,983 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:32,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:32,987 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:32,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:32,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:32,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:32,987 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:32,987 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:32,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:32,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:33,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024253003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024253003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024253006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024253008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024253009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,088 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:33,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:33,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:33,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:33,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:33,091 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:33,091 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:33,091 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:33,091 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:33,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,093 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:33,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:33,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,094 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:33,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:33,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:33,095 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,095 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:33,095 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:33,095 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:33,097 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab04348[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:33,099 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:33,099 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,100 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,101 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:33,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,103 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:33,103 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:33,103 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:33,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:33,106 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:33,106 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:33,106 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:33,106 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:33,106 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:33,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:33,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,107 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:33,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:33,109 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,111 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:33,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,113 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:33,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,113 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,113 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:33,114 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:33,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024253207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024253207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024253210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,214 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:33,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024253212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024253213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:33,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:33,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:33,217 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:33,217 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:33,217 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:33,217 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:33,218 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,218 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,219 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:33,219 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:33,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:33,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:33,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:33,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:33,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:33,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:33,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:33,221 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:33,221 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,221 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:33,222 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:33,222 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:33,224 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4308bda0[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:33,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:33,226 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:33,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:33,228 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:33,228 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,228 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:33,230 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:33,233 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:33,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:33,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:33,234 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:33,234 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:33,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:33,238 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:33,239 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:33,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,240 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,252 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:33,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:33,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:33,253 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:33,253 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:33,253 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:33,301 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d9ee614783d547e4afc1d0b4802bc0ea to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d9ee614783d547e4afc1d0b4802bc0ea 2023-04-20 20:56:33,306 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1497, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/25cb61998af84aafa6e4d4972c01322f 2023-04-20 20:56:33,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1497 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/ca185c6f84f84391afc130359bd03f1a 2023-04-20 20:56:33,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1497 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/ed5a6cb9a97546728136bcaea1f99537 2023-04-20 20:56:33,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/25cb61998af84aafa6e4d4972c01322f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/25cb61998af84aafa6e4d4972c01322f 2023-04-20 20:56:33,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/25cb61998af84aafa6e4d4972c01322f, entries=200, sequenceid=1497, filesize=32.3 K 2023-04-20 20:56:33,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/ca185c6f84f84391afc130359bd03f1a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ca185c6f84f84391afc130359bd03f1a 2023-04-20 20:56:33,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ca185c6f84f84391afc130359bd03f1a, entries=150, sequenceid=1497, filesize=11.9 K 2023-04-20 20:56:33,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/ed5a6cb9a97546728136bcaea1f99537 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ed5a6cb9a97546728136bcaea1f99537 2023-04-20 20:56:33,353 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:33,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:33,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:33,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:33,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:33,357 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:33,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:33,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:33,357 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:33,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ed5a6cb9a97546728136bcaea1f99537, entries=150, sequenceid=1497, filesize=11.9 K 2023-04-20 20:56:33,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,358 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 819071c32df35da7c00fe2675f028bf5 in 469ms, sequenceid=1497, compaction requested=true 2023-04-20 20:56:33,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:33,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:33,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:33,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:33,358 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:56:33,358 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:56:33,359 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,359 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:33,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:33,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,360 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,360 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:56:33,360 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:33,360 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,360 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/99a44a6830b640299a5fff4b79e93acb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/003ec7aea6f443659db9b8956f567866, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3c48fded5cfc4ab59040f1c0c3213aa4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b9285ed8adf744aead22e911eb60b615, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/0f5cad6420e74d86b8cf7499e2e36819, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ca185c6f84f84391afc130359bd03f1a] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=72.0 K 2023-04-20 20:56:33,360 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163896 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:56:33,360 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/A is initiating minor compaction (all files) 2023-04-20 20:56:33,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:33,361 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/A in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,361 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 99a44a6830b640299a5fff4b79e93acb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682024187694 2023-04-20 20:56:33,361 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0b6bce97acf84c0a9d24708b59795782, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2b883f3a48ab4f12bca74f87ba8250f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d20a7cd5588646b2a767a262a7e61b36, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/24d9438c0890458286dc421e3d150164, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fb2610d32e734357bc3b4d109d6474e8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/25cb61998af84aafa6e4d4972c01322f] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=160.1 K 2023-04-20 20:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:33,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:33,361 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0b6bce97acf84c0a9d24708b59795782, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682024187694 2023-04-20 20:56:33,361 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 003ec7aea6f443659db9b8956f567866, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1427, earliestPutTs=1682024189350 2023-04-20 20:56:33,362 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2b883f3a48ab4f12bca74f87ba8250f7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1427, earliestPutTs=1682024189350 2023-04-20 20:56:33,362 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3c48fded5cfc4ab59040f1c0c3213aa4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1447, earliestPutTs=1682024189387 2023-04-20 20:56:33,362 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d20a7cd5588646b2a767a262a7e61b36, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1447, earliestPutTs=1682024189387 2023-04-20 20:56:33,362 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting b9285ed8adf744aead22e911eb60b615, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1682024190522 2023-04-20 20:56:33,363 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 24d9438c0890458286dc421e3d150164, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1682024190522 2023-04-20 20:56:33,363 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting fb2610d32e734357bc3b4d109d6474e8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1479, earliestPutTs=1682024191145 2023-04-20 20:56:33,363 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 0f5cad6420e74d86b8cf7499e2e36819, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1479, earliestPutTs=1682024191145 2023-04-20 20:56:33,363 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 25cb61998af84aafa6e4d4972c01322f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1497, earliestPutTs=1682024192276 2023-04-20 20:56:33,363 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ca185c6f84f84391afc130359bd03f1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1497, earliestPutTs=1682024192277 2023-04-20 20:56:33,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:33,365 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,365 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:33,365 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 20:56:33,365 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:33,366 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:33,366 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:33,366 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:33,366 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:33,366 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:33,379 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#610 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:33,394 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#A#compaction#611 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:33,408 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/5bee712c63354234ac33357ed08fc6cc as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5bee712c63354234ac33357ed08fc6cc 2023-04-20 20:56:33,412 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into 5bee712c63354234ac33357ed08fc6cc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:33,412 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:33,412 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=10, startTime=1682024193358; duration=0sec 2023-04-20 20:56:33,412 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:33,412 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 20:56:33,414 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 20:56:33,414 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/C is initiating minor compaction (all files) 2023-04-20 20:56:33,415 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/C in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:33,415 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/cc27dabe4434407ead1de06183cca8a8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/530c19bb0a614739a37a0fd461722697, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9315636958ba4bffbf961bbe75f86db2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/30d318a5665044b7a224e9f765f9aaa4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/76a16bf719414227941e5b5a4b1eb35a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ed5a6cb9a97546728136bcaea1f99537] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=72.0 K 2023-04-20 20:56:33,415 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting cc27dabe4434407ead1de06183cca8a8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682024187694 2023-04-20 20:56:33,415 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 530c19bb0a614739a37a0fd461722697, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1427, earliestPutTs=1682024189350 2023-04-20 20:56:33,415 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 9315636958ba4bffbf961bbe75f86db2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1447, earliestPutTs=1682024189387 2023-04-20 20:56:33,416 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 30d318a5665044b7a224e9f765f9aaa4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1682024190522 2023-04-20 20:56:33,416 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 76a16bf719414227941e5b5a4b1eb35a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1479, earliestPutTs=1682024191145 2023-04-20 20:56:33,416 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting ed5a6cb9a97546728136bcaea1f99537, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1497, earliestPutTs=1682024192277 2023-04-20 20:56:33,427 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#C#compaction#613 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:33,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:33,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:33,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024253527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024253528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024253528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024253529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024253529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024253632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024253633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024253633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024253635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024253635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,819 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d007738f9aa04728ace206a43883793d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d007738f9aa04728ace206a43883793d 2023-04-20 20:56:33,823 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1511, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/0434f3d3dbd54b78a3aff4fadfc48535 2023-04-20 20:56:33,832 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1511 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/c3f6300f13de4ff5afc173f5eb4a5c13 2023-04-20 20:56:33,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024253835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024253838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024253838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024253840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:33,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024253841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:33,847 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/66397155b68d428dbec9938190cdc27b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/66397155b68d428dbec9938190cdc27b 2023-04-20 20:56:33,851 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/d0132fc7b67947dda24dbe2dd5cc09c1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d0132fc7b67947dda24dbe2dd5cc09c1 2023-04-20 20:56:33,852 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/C of 819071c32df35da7c00fe2675f028bf5 into 66397155b68d428dbec9938190cdc27b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:33,852 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:33,852 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/C, priority=10, startTime=1682024193358; duration=0sec 2023-04-20 20:56:33,853 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:33,855 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/A of 819071c32df35da7c00fe2675f028bf5 into d0132fc7b67947dda24dbe2dd5cc09c1(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:33,855 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:33,855 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/A, priority=10, startTime=1682024193358; duration=0sec 2023-04-20 20:56:33,856 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:34,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024254137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024254141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024254143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024254144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024254146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,243 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1511 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/e0196d4ad8d645babcc8f40d1ec19dc5 2023-04-20 20:56:34,246 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/0434f3d3dbd54b78a3aff4fadfc48535 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0434f3d3dbd54b78a3aff4fadfc48535 2023-04-20 20:56:34,250 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0434f3d3dbd54b78a3aff4fadfc48535, entries=150, sequenceid=1511, filesize=25.4 K 2023-04-20 20:56:34,251 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/c3f6300f13de4ff5afc173f5eb4a5c13 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/c3f6300f13de4ff5afc173f5eb4a5c13 2023-04-20 20:56:34,254 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/c3f6300f13de4ff5afc173f5eb4a5c13, entries=150, sequenceid=1511, filesize=11.9 K 2023-04-20 20:56:34,254 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/e0196d4ad8d645babcc8f40d1ec19dc5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e0196d4ad8d645babcc8f40d1ec19dc5 2023-04-20 20:56:34,258 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e0196d4ad8d645babcc8f40d1ec19dc5, entries=150, sequenceid=1511, filesize=11.9 K 2023-04-20 20:56:34,258 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 819071c32df35da7c00fe2675f028bf5 in 893ms, sequenceid=1511, compaction requested=false 2023-04-20 20:56:34,258 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:34,258 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:34,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:34,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:34,259 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:34,260 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,261 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69fe7dda[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:34,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:34,262 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,265 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:34,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,265 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:34,267 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:34,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,267 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:34,270 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:34,270 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:34,270 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:34,270 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:34,270 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:34,273 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,273 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,273 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,273 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,275 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,277 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,277 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,277 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,278 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,278 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:34,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:34,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,278 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:34,278 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:34,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:34,378 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:34,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:34,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:34,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:34,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:34,382 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:34,382 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,382 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:34,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:34,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,383 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,385 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,385 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,386 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:34,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:34,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:34,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:34,387 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:34,387 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,387 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:34,387 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 20:56:34,387 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:34,387 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:34,388 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:34,388 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:34,388 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:34,388 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:34,401 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304200c4f614c94094c8e95a5f60affbe2b89 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200c4f614c94094c8e95a5f60affbe2b89 2023-04-20 20:56:34,409 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1532, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/44f4684fda0242ec9e61b55b1be90088 2023-04-20 20:56:34,417 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1532 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2690dd63670047daa3447424ea3d587a 2023-04-20 20:56:34,434 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1532 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/29b7705b767a43d4b96908b69c7550c1 2023-04-20 20:56:34,437 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/44f4684fda0242ec9e61b55b1be90088 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/44f4684fda0242ec9e61b55b1be90088 2023-04-20 20:56:34,441 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/44f4684fda0242ec9e61b55b1be90088, entries=150, sequenceid=1532, filesize=25.4 K 2023-04-20 20:56:34,442 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2690dd63670047daa3447424ea3d587a as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2690dd63670047daa3447424ea3d587a 2023-04-20 20:56:34,445 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2690dd63670047daa3447424ea3d587a, entries=150, sequenceid=1532, filesize=11.9 K 2023-04-20 20:56:34,445 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/29b7705b767a43d4b96908b69c7550c1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/29b7705b767a43d4b96908b69c7550c1 2023-04-20 20:56:34,448 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/29b7705b767a43d4b96908b69c7550c1, entries=150, sequenceid=1532, filesize=11.9 K 2023-04-20 20:56:34,449 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 819071c32df35da7c00fe2675f028bf5 in 62ms, sequenceid=1532, compaction requested=true 2023-04-20 20:56:34,449 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:34,449 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:34,451 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,451 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:34,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403e2f31[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:34,453 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:34,453 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,455 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:34,455 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,455 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:34,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:34,455 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:34,456 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:34,457 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:34,459 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:34,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:34,459 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:34,460 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:34,460 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:34,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,462 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,463 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,464 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:34,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:34,467 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:34,467 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,467 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:34,467 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,567 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:34,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:34,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:34,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:34,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:34,571 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:34,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:34,571 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:34,572 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,572 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,573 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,573 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,574 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,574 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:34,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:34,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:34,575 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,575 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:34,575 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:34,575 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:34,576 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:34,578 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@184c9e7d[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:34,580 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:34,580 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,587 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,587 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:34,587 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:34,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,587 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:34,587 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,589 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:34,592 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:34,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:34,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:34,592 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:34,592 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:34,593 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,593 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,594 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,595 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,596 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,602 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,602 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,603 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:34,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:34,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,603 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,603 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:34,604 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:34,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:34,669 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:34,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:34,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:34,671 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:34,671 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:34,671 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:34,671 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:34,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682024194685, max timestamp=1682024194692Num uniques 859; 2023-04-20 20:56:34,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682024194685, max timestamp=1682024194692Num uniques 859; 2023-04-20 20:56:34,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682024194685, max timestamp=1682024194692Num uniques 859; 2023-04-20 20:56:34,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:56:34,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:56:34,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 20:56:34,704 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:34,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:34,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:34,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024254705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024254706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024254707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:34,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:34,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024254708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,711 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:34,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:34,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:34,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,711 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,713 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,713 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024254710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:34,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:34,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:34,714 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:34,714 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:34,715 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,715 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:34,715 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:34,716 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:34,717 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eafc569[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:34,720 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:34,720 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,721 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:34,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:34,722 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:34,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,722 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:34,724 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:34,727 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:34,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:34,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:34,727 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:34,727 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:34,729 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,729 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,729 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,730 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,731 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,733 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:34,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:34,734 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,734 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:34,734 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:34,734 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:34,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024254811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024254811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024254812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024254812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024254815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,834 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:34,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:34,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:34,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:34,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:34,838 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:34,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:34,838 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:34,839 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,839 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,840 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,840 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:34,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:34,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:34,845 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:34,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a57bcb[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:34,847 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:34,847 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,849 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:34,849 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,849 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:34,850 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,850 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:34,850 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,851 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:34,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:34,853 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:34,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:34,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:34,853 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:34,853 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:34,855 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,855 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,855 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,856 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,857 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,861 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:34,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:34,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,861 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,862 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:34,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,862 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,862 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:34,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:34,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,962 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:34,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:34,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:34,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:34,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:34,966 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:34,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:34,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:34,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,966 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,968 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:34,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:34,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:34,970 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,970 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:34,970 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:34,970 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:34,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35d39bc9[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:34,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:34,974 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,975 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:34,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:34,975 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:34,975 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,976 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:34,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:34,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:34,981 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:34,981 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:34,981 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:34,981 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:34,981 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:34,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,982 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,983 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,983 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,984 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:34,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:34,987 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:34,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:34,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:34,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:34,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:34,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:34,988 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:34,988 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:34,988 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:34,988 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024255016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024255016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024255017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024255017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024255017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,085 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420ab7e2b57ca1342c8be1264c1b1b2c51b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ab7e2b57ca1342c8be1264c1b1b2c51b 2023-04-20 20:56:35,088 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:35,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:35,091 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1544, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/e96f35cf969447edb2b0d3a94c42d72d 2023-04-20 20:56:35,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:35,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:35,092 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:35,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:35,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:35,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,093 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,103 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,103 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,104 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:35,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:35,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:35,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:35,106 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,106 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:35,106 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1544 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2a886d0371b14376aa0d07715c9febb9 2023-04-20 20:56:35,106 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:35,106 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:35,109 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:35,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49105478[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:35,113 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:35,113 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,114 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,114 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:35,114 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:35,114 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:35,114 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,115 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:35,117 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1544 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/5e101497b13a46019cb0d9e27edf3bc2 2023-04-20 20:56:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,118 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:35,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:35,120 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:35,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:35,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:35,120 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:35,120 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:35,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/e96f35cf969447edb2b0d3a94c42d72d as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e96f35cf969447edb2b0d3a94c42d72d 2023-04-20 20:56:35,122 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,122 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,122 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,123 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,124 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e96f35cf969447edb2b0d3a94c42d72d, entries=400, sequenceid=1544, filesize=59.8 K 2023-04-20 20:56:35,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/2a886d0371b14376aa0d07715c9febb9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2a886d0371b14376aa0d07715c9febb9 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,127 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:35,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,127 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,128 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,128 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:35,128 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:35,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2a886d0371b14376aa0d07715c9febb9, entries=150, sequenceid=1544, filesize=11.9 K 2023-04-20 20:56:35,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/5e101497b13a46019cb0d9e27edf3bc2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/5e101497b13a46019cb0d9e27edf3bc2 2023-04-20 20:56:35,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/5e101497b13a46019cb0d9e27edf3bc2, entries=150, sequenceid=1544, filesize=11.9 K 2023-04-20 20:56:35,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 819071c32df35da7c00fe2675f028bf5 in 465ms, sequenceid=1544, compaction requested=true 2023-04-20 20:56:35,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:35,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:35,134 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:35,134 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 20:56:35,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:35,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:35,135 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 140176 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:35,135 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 20:56:35,135 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/A is initiating minor compaction (all files) 2023-04-20 20:56:35,135 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:35,135 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/A in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,135 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,135 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d0132fc7b67947dda24dbe2dd5cc09c1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0434f3d3dbd54b78a3aff4fadfc48535, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/44f4684fda0242ec9e61b55b1be90088, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e96f35cf969447edb2b0d3a94c42d72d] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=136.9 K 2023-04-20 20:56:35,135 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5bee712c63354234ac33357ed08fc6cc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/c3f6300f13de4ff5afc173f5eb4a5c13, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2690dd63670047daa3447424ea3d587a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2a886d0371b14376aa0d07715c9febb9] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=48.4 K 2023-04-20 20:56:35,136 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting d0132fc7b67947dda24dbe2dd5cc09c1, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=1497, earliestPutTs=1682024192277 2023-04-20 20:56:35,136 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5bee712c63354234ac33357ed08fc6cc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1497, earliestPutTs=1682024192277 2023-04-20 20:56:35,136 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 0434f3d3dbd54b78a3aff4fadfc48535, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1511, earliestPutTs=1682024192900 2023-04-20 20:56:35,136 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting c3f6300f13de4ff5afc173f5eb4a5c13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1511, earliestPutTs=1682024192900 2023-04-20 20:56:35,136 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 44f4684fda0242ec9e61b55b1be90088, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1682024193523 2023-04-20 20:56:35,137 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2690dd63670047daa3447424ea3d587a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1682024193523 2023-04-20 20:56:35,137 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting e96f35cf969447edb2b0d3a94c42d72d, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1682024194656 2023-04-20 20:56:35,137 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 2a886d0371b14376aa0d07715c9febb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1682024194665 2023-04-20 20:56:35,143 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#623 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:35,156 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#A#compaction#622 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:35,228 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:35,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:35,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:35,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:35,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:35,231 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:35,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:35,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:35,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,231 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,233 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,233 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:35,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:35,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:35,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:35,235 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:35,235 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:35,255 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304209574d7c513524f9eb2ea34bd9dc00a67 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209574d7c513524f9eb2ea34bd9dc00a67 2023-04-20 20:56:35,264 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1578, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/30f3332b312043a48eaca080c6f4bdef 2023-04-20 20:56:35,273 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1578 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/d2a71b5807b8461cb389a2601667fb7b 2023-04-20 20:56:35,282 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1578 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/3c01463fedde4d29853ffd78587d3b27 2023-04-20 20:56:35,285 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/30f3332b312043a48eaca080c6f4bdef as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/30f3332b312043a48eaca080c6f4bdef 2023-04-20 20:56:35,289 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/30f3332b312043a48eaca080c6f4bdef, entries=150, sequenceid=1578, filesize=25.4 K 2023-04-20 20:56:35,289 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/d2a71b5807b8461cb389a2601667fb7b as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/d2a71b5807b8461cb389a2601667fb7b 2023-04-20 20:56:35,292 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/d2a71b5807b8461cb389a2601667fb7b, entries=150, sequenceid=1578, filesize=11.9 K 2023-04-20 20:56:35,293 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/3c01463fedde4d29853ffd78587d3b27 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3c01463fedde4d29853ffd78587d3b27 2023-04-20 20:56:35,296 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3c01463fedde4d29853ffd78587d3b27, entries=150, sequenceid=1578, filesize=11.9 K 2023-04-20 20:56:35,296 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=0 B/0 for 819071c32df35da7c00fe2675f028bf5 in 61ms, sequenceid=1578, compaction requested=true 2023-04-20 20:56:35,296 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:35,296 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,296 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:35,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:35,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:35,297 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:35,299 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,300 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:35,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f454be[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:35,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:35,301 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,303 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:35,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,303 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:35,305 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:35,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:35,308 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:35,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:35,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:35,308 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:35,308 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:35,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,310 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,310 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,312 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,315 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:35,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:35,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,315 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:35,315 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,315 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:35,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 20:56:35,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:35,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:35,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:35,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:35,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:35,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:35,377 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420b64baa1a14e542ed82afc9f86e3eb94c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b64baa1a14e542ed82afc9f86e3eb94c 2023-04-20 20:56:35,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024255383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024255383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024255384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024255384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024195381, max timestamp=1682024195387Num uniques 859; 2023-04-20 20:56:35,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024195381, max timestamp=1682024195387Num uniques 859; 2023-04-20 20:56:35,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682024195381, max timestamp=1682024195387Num uniques 859; 2023-04-20 20:56:35,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:35,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:35,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 20:56:35,416 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:35,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:35,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:35,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:35,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:35,418 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:35,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:35,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:35,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,419 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,420 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,420 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,421 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,421 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:35,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:35,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:35,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:35,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:35,422 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:35,423 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,423 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:35,423 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:35,423 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:35,424 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,425 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a00968[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:35,426 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:35,426 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,427 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,427 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,428 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:35,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:35,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:35,428 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:35,430 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:35,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,430 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:35,433 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:35,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:35,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:35,433 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:35,433 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:35,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,434 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,435 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,435 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,436 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:35,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:35,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,439 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,439 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:35,439 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,439 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024255451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024255489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024255490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024255491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024255491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,539 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:35,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:35,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:35,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:35,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:35,543 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:35,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:35,543 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:35,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,544 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,545 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,545 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,546 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:35,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:35,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:35,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:35,547 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:35,554 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f04d7a6[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:35,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:35,556 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,558 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,558 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:35,558 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:35,558 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:35,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024255556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,559 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:35,561 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:35,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,561 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,562 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/23b547c45efd4ed294c8834d26111ee4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/23b547c45efd4ed294c8834d26111ee4 2023-04-20 20:56:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:35,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:35,565 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:35,565 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:35,565 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:35,565 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:35,565 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:35,566 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into 23b547c45efd4ed294c8834d26111ee4(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:35,566 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:35,566 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=12, startTime=1682024195134; duration=0sec 2023-04-20 20:56:35,566 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:35,567 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 20:56:35,568 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 20:56:35,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,568 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/C is initiating minor compaction (all files) 2023-04-20 20:56:35,568 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,568 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/C in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,569 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/66397155b68d428dbec9938190cdc27b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e0196d4ad8d645babcc8f40d1ec19dc5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/29b7705b767a43d4b96908b69c7550c1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/5e101497b13a46019cb0d9e27edf3bc2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3c01463fedde4d29853ffd78587d3b27] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=60.3 K 2023-04-20 20:56:35,569 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,569 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 66397155b68d428dbec9938190cdc27b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1497, earliestPutTs=1682024192277 2023-04-20 20:56:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,570 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting e0196d4ad8d645babcc8f40d1ec19dc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1511, earliestPutTs=1682024192900 2023-04-20 20:56:35,570 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 29b7705b767a43d4b96908b69c7550c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1682024193523 2023-04-20 20:56:35,570 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,571 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 5e101497b13a46019cb0d9e27edf3bc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1682024194665 2023-04-20 20:56:35,572 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,572 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 3c01463fedde4d29853ffd78587d3b27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1578, earliestPutTs=1682024194702 2023-04-20 20:56:35,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,576 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:35,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:35,576 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,577 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,577 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,577 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:35,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,578 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:35,578 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,583 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#C#compaction#628 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:35,591 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/55e8d0a22ccc432a95ff5d6354925a68 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55e8d0a22ccc432a95ff5d6354925a68 2023-04-20 20:56:35,596 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/C of 819071c32df35da7c00fe2675f028bf5 into 55e8d0a22ccc432a95ff5d6354925a68(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:35,596 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:35,596 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/C, priority=11, startTime=1682024195134; duration=0sec 2023-04-20 20:56:35,596 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:35,678 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:35,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:35,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:35,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:35,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:35,682 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:35,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:35,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:35,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,682 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,683 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,684 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,684 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:35,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:35,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:35,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:35,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:35,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:35,685 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:35,685 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,686 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:35,686 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:35,686 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:35,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:35,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:35,686 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:35,687 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:35,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647ce5aa[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:35,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:35,689 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,691 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:35,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:35,691 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:35,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,691 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:35,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:35,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,693 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,693 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:35,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:35,696 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:35,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:35,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:35,696 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:35,696 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:35,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024255694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024255694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024255694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024255695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,698 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,699 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,700 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,702 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,703 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:35,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,703 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,703 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:35,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:35,703 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024255761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,786 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1589, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/2186ffcace36427da39cf6a7b6eaab84 2023-04-20 20:56:35,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1589 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/8a75688a5935454fa553484341decb4e 2023-04-20 20:56:35,803 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:35,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:35,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:35,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:35,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:35,807 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:35,807 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,807 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:35,807 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:35,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,808 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,810 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,810 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:35,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:35,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:35,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:35,811 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:35,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:35,811 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:35,812 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:35,813 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,813 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa61c3f[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:35,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:35,815 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,816 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:35,817 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,817 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:35,818 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:35,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,818 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:35,821 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:35,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:35,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:35,821 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:35,821 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:35,834 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,834 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,834 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,836 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,837 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,841 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:35,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:35,841 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,842 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:35,842 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,942 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:35,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:35,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:35,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:35,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:35,946 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:35,946 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,946 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:35,946 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:35,947 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,947 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,949 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:35,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:35,951 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,951 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:35,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:35,952 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:35,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:35,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:35,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:35,952 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:35,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,952 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:35,953 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,953 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@694ba3ea[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:35,955 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:35,955 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,958 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:35,958 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,958 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:35,961 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:35,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:35,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:35,964 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:35,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:35,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:35,964 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:35,964 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:35,966 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,966 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,966 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:35,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:35,967 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:35,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,968 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:35,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:35,970 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:35,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:35,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:35,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:35,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:35,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:35,970 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:35,971 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:35,971 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:35,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:35,971 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:35,985 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/fa9dd411c3d846bcab7acc89e7342738 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fa9dd411c3d846bcab7acc89e7342738 2023-04-20 20:56:35,989 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/A of 819071c32df35da7c00fe2675f028bf5 into fa9dd411c3d846bcab7acc89e7342738(size=26.3 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:35,989 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:35,989 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/A, priority=12, startTime=1682024195134; duration=0sec 2023-04-20 20:56:35,989 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:35,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024255998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:35,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:35,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024255998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024255999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024256000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024256065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,071 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:36,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:36,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:36,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:36,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:36,075 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:36,075 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:36,075 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:36,075 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:36,075 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,075 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,080 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:36,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:36,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:36,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,081 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:36,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:36,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:36,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:36,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:36,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:36,082 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:36,083 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:36,083 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:36,083 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:36,083 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:36,083 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,083 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:36,083 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:36,084 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:36,084 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:36,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,085 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:36,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:36,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607be082[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:36,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:36,086 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,087 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:36,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:36,088 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:36,088 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,088 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:36,089 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:36,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:36,092 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:36,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:36,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:36,092 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:36,092 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:36,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:36,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:36,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,094 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:36,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:36,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:36,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:36,095 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:36,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,096 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:36,098 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:36,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:36,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,099 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,099 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:36,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:36,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:36,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:36,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:36,100 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:36,100 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:36,200 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:36,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:36,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:36,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:36,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:36,203 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:36,203 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:36,203 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:36,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:36,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,204 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,205 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:36,205 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:36,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,206 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:36,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1589 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/b4fb5e9bfb244fc69e99a34a10fcecf3 2023-04-20 20:56:36,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:36,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:36,208 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:36,208 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:36,208 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:36,208 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:36,210 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/2186ffcace36427da39cf6a7b6eaab84 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2186ffcace36427da39cf6a7b6eaab84 2023-04-20 20:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:36,211 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,211 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63a90892[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:36,212 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:36,212 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2186ffcace36427da39cf6a7b6eaab84, entries=400, sequenceid=1589, filesize=59.8 K 2023-04-20 20:56:36,214 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/8a75688a5935454fa553484341decb4e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/8a75688a5935454fa553484341decb4e 2023-04-20 20:56:36,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,215 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:36,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:36,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:36,216 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,216 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:36,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/8a75688a5935454fa553484341decb4e, entries=150, sequenceid=1589, filesize=11.9 K 2023-04-20 20:56:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/b4fb5e9bfb244fc69e99a34a10fcecf3 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b4fb5e9bfb244fc69e99a34a10fcecf3 2023-04-20 20:56:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:36,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:36,220 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:36,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:36,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:36,220 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:36,220 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:36,221 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,222 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:36,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,222 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:36,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:36,223 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b4fb5e9bfb244fc69e99a34a10fcecf3, entries=150, sequenceid=1589, filesize=11.9 K 2023-04-20 20:56:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:36,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 819071c32df35da7c00fe2675f028bf5 in 872ms, sequenceid=1589, compaction requested=true 2023-04-20 20:56:36,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:36,224 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:36,224 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 20:56:36,224 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:36,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:36,225 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 114266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:36,225 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 20:56:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:36,225 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/A is initiating minor compaction (all files) 2023-04-20 20:56:36,225 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/A in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:36,225 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fa9dd411c3d846bcab7acc89e7342738, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/30f3332b312043a48eaca080c6f4bdef, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2186ffcace36427da39cf6a7b6eaab84] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=111.6 K 2023-04-20 20:56:36,225 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1912): 819071c32df35da7c00fe2675f028bf5/B is initiating minor compaction (all files) 2023-04-20 20:56:36,225 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 819071c32df35da7c00fe2675f028bf5/B in TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:36,225 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/23b547c45efd4ed294c8834d26111ee4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/d2a71b5807b8461cb389a2601667fb7b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/8a75688a5935454fa553484341decb4e] into tmpdir=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp, totalSize=36.6 K 2023-04-20 20:56:36,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,226 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting fa9dd411c3d846bcab7acc89e7342738, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1682024194665 2023-04-20 20:56:36,227 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 23b547c45efd4ed294c8834d26111ee4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1682024194665 2023-04-20 20:56:36,227 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 30f3332b312043a48eaca080c6f4bdef, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1578, earliestPutTs=1682024194702 2023-04-20 20:56:36,227 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting d2a71b5807b8461cb389a2601667fb7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1578, earliestPutTs=1682024194702 2023-04-20 20:56:36,227 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] compactions.Compactor(207): Compacting 2186ffcace36427da39cf6a7b6eaab84, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1589, earliestPutTs=1682024195336 2023-04-20 20:56:36,227 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.Compactor(207): Compacting 8a75688a5935454fa553484341decb4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1589, earliestPutTs=1682024195350 2023-04-20 20:56:36,227 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,228 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:36,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:36,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:36,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:36,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:36,228 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,228 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:36,228 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:36,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:36,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:36,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:36,234 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#B#compaction#632 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:36,237 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] throttle.PressureAwareThroughputController(145): 819071c32df35da7c00fe2675f028bf5#A#compaction#631 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 20:56:36,248 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/5355d07fa28d475fa8a4f867e8482ec1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5355d07fa28d475fa8a4f867e8482ec1 2023-04-20 20:56:36,253 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/B of 819071c32df35da7c00fe2675f028bf5 into 5355d07fa28d475fa8a4f867e8482ec1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:36,253 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:36,253 INFO [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/B, priority=13, startTime=1682024196224; duration=0sec 2023-04-20 20:56:36,253 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 20:56:36,253 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-20 20:56:36,254 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 20:56:36,254 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 20:56:36,254 DEBUG [RS:0;jenkins-hbase4:39223-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. because compaction request was cancelled 2023-04-20 20:56:36,328 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:36,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:36,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-20 20:56:36,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 20:56:36,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 20:56:36,332 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 20:56:36,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:36,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 20:56:36,332 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 20:56:36,333 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,333 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,335 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,335 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:36,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:36,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:36,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,336 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 20:56:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 20:56:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:36,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 20:56:36,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 20:56:36,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 20:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 20:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 20:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 20:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 20:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 20:56:36,337 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:36,337 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. started... 2023-04-20 20:56:36,338 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-20 20:56:36,338 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:36,338 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:36,338 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:36,339 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:36,339 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:36,339 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 20:56:36,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:36,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. as already flushing 2023-04-20 20:56:36,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024256507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024256507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024256508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024256510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024256571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,578 DEBUG [Thread-4013] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:62055 2023-04-20 20:56:36,578 DEBUG [Thread-4013] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:36,579 DEBUG [Thread-4019] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eedd898 to 127.0.0.1:62055 2023-04-20 20:56:36,579 DEBUG [Thread-4019] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:36,579 DEBUG [Thread-4015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:62055 2023-04-20 20:56:36,579 DEBUG [Thread-4015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:36,582 DEBUG [Thread-4011] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:62055 2023-04-20 20:56:36,582 DEBUG [Thread-4011] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:36,583 DEBUG [Thread-4017] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:62055 2023-04-20 20:56:36,583 DEBUG [Thread-4017] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:36,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024256610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024256611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024256612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024256613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,656 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/d52b7bf923f5406e8118db4912e855ae as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d52b7bf923f5406e8118db4912e855ae 2023-04-20 20:56:36,659 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 819071c32df35da7c00fe2675f028bf5/A of 819071c32df35da7c00fe2675f028bf5 into d52b7bf923f5406e8118db4912e855ae(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 20:56:36,659 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:36,659 INFO [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5., storeName=819071c32df35da7c00fe2675f028bf5/A, priority=13, startTime=1682024196224; duration=0sec 2023-04-20 20:56:36,659 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:36,760 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420c52c18a5bc794e6699de1181a46ca478 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c52c18a5bc794e6699de1181a46ca478 2023-04-20 20:56:36,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024256812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024256813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024256813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:36,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:36,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024256815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024257114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024257114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024257115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024257118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,164 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1629, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/a911c6d14a664b7985abc71f7bff57b9 2023-04-20 20:56:37,573 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1629 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/ceff4e37a0f84336a9a8d448c65ced84 2023-04-20 20:56:37,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59536 deadline: 1682024257576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39223] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59482 deadline: 1682024257618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59566 deadline: 1682024257618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59522 deadline: 1682024257620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 20:56:37,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59472 deadline: 1682024257620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,981 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1629 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/ab9e8b457e744c32b82e1247d997832f 2023-04-20 20:56:37,984 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/a911c6d14a664b7985abc71f7bff57b9 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/a911c6d14a664b7985abc71f7bff57b9 2023-04-20 20:56:37,987 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/a911c6d14a664b7985abc71f7bff57b9, entries=150, sequenceid=1629, filesize=25.4 K 2023-04-20 20:56:37,988 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/ceff4e37a0f84336a9a8d448c65ced84 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ceff4e37a0f84336a9a8d448c65ced84 2023-04-20 20:56:37,991 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ceff4e37a0f84336a9a8d448c65ced84, entries=150, sequenceid=1629, filesize=11.9 K 2023-04-20 20:56:37,991 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/ab9e8b457e744c32b82e1247d997832f as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ab9e8b457e744c32b82e1247d997832f 2023-04-20 20:56:37,994 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ab9e8b457e744c32b82e1247d997832f, entries=150, sequenceid=1629, filesize=11.9 K 2023-04-20 20:56:37,995 INFO [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 819071c32df35da7c00fe2675f028bf5 in 1657ms, sequenceid=1629, compaction requested=true 2023-04-20 20:56:37,995 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:37,995 DEBUG [rs(jenkins-hbase4.apache.org,39223,1682024020002)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:37,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 20:56:37,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 20:56:37,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 20:56:37,995 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 20:56:37,997 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 20:56:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:37,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39223,1682024020002' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 20:56:37,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c31b73c[Count = 0] remaining members to acquire global barrier 2023-04-20 20:56:37,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 20:56:37,998 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:38,000 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:38,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:38,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:38,000 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,000 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 20:56:38,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 20:56:38,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 20:56:38,000 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39223,1682024020002' in zk 2023-04-20 20:56:38,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 20:56:38,007 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:38,007 DEBUG [member: 'jenkins-hbase4.apache.org,39223,1682024020002' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 20:56:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39223,1682024020002': 2023-04-20 20:56:38,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39223,1682024020002' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 20:56:38,009 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 20:56:38,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 20:56:38,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 20:56:38,009 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 20:56:38,009 INFO [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 20:56:38,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:38,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:38,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:38,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:38,010 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:38,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 20:56:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 20:56:38,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:38,011 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 20:56:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 20:56:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:38,012 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 20:56:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 20:56:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:38,014 DEBUG [(jenkins-hbase4.apache.org,44655,1682024018782)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 20:56:38,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 20:56:38,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 20:56:38,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 20:56:38,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 20:56:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 20:56:38,014 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 20:56:38,014 DEBUG [Thread-4010] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 20:56:38,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 20:56:38,015 DEBUG [Thread-4010] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 20:56:38,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 20:56:38,115 DEBUG [Thread-4010] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 20:56:38,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 20:56:38,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39223] regionserver.HRegion(9158): Flush requested on 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:38,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 20:56:38,627 DEBUG [Thread-4008] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:62055 2023-04-20 20:56:38,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:38,627 DEBUG [Thread-4008] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:38,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:38,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:38,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:38,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:38,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:38,628 DEBUG [Thread-4000] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:62055 2023-04-20 20:56:38,628 DEBUG [Thread-4000] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:38,629 DEBUG [Thread-4002] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:62055 2023-04-20 20:56:38,629 DEBUG [Thread-4002] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:38,629 DEBUG [Thread-4006] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:62055 2023-04-20 20:56:38,629 DEBUG [Thread-4006] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:39,037 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420c331866ff22f44aaa20b29d502d49400 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c331866ff22f44aaa20b29d502d49400 2023-04-20 20:56:39,441 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1642, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/c44e6b5d31ce4d949378c3db925897a2 2023-04-20 20:56:39,588 DEBUG [Thread-4004] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:62055 2023-04-20 20:56:39,588 DEBUG [Thread-4004] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:39,588 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 274 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 301 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 327 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 324 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(394): wrote 307 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 3998 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 3993 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 3905 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 4016 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(398): read 3994 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 20:56:39,589 INFO [Listener at localhost/45619] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 20:56:39,589 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:62055 2023-04-20 20:56:39,589 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:39,590 INFO [Listener at localhost/45619] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 20:56:39,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-20 20:56:39,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:39,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 20:56:39,593 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024199593"}]},"ts":"1682024199593"} 2023-04-20 20:56:39,594 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 20:56:39,595 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 20:56:39,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, UNASSIGN}] 2023-04-20 20:56:39,597 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, UNASSIGN 2023-04-20 20:56:39,597 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:39,598 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024199597"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682024199597"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682024199597"}]},"ts":"1682024199597"} 2023-04-20 20:56:39,598 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002}] 2023-04-20 20:56:39,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 20:56:39,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:39,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 819071c32df35da7c00fe2675f028bf5, disabling compactions & flushes 2023-04-20 20:56:39,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:39,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1642 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/55e60db9097c4e9c9fac6aaf1592a000 2023-04-20 20:56:39,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 20:56:40,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 20:56:40,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1642 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/3eb6a263700f4453b7ba28f6aff401bd 2023-04-20 20:56:40,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/c44e6b5d31ce4d949378c3db925897a2 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c44e6b5d31ce4d949378c3db925897a2 2023-04-20 20:56:40,264 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c44e6b5d31ce4d949378c3db925897a2, entries=150, sequenceid=1642, filesize=25.4 K 2023-04-20 20:56:40,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/55e60db9097c4e9c9fac6aaf1592a000 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/55e60db9097c4e9c9fac6aaf1592a000 2023-04-20 20:56:40,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/55e60db9097c4e9c9fac6aaf1592a000, entries=150, sequenceid=1642, filesize=11.9 K 2023-04-20 20:56:40,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/3eb6a263700f4453b7ba28f6aff401bd as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3eb6a263700f4453b7ba28f6aff401bd 2023-04-20 20:56:40,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3eb6a263700f4453b7ba28f6aff401bd, entries=150, sequenceid=1642, filesize=11.9 K 2023-04-20 20:56:40,272 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 819071c32df35da7c00fe2675f028bf5 in 1646ms, sequenceid=1642, compaction requested=true 2023-04-20 20:56:40,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:40,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:40,272 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. because compaction request was cancelled 2023-04-20 20:56:40,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:40,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:40,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. after waiting 0 ms 2023-04-20 20:56:40,272 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. because compaction request was cancelled 2023-04-20 20:56:40,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:40,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:40,272 DEBUG [RS:0;jenkins-hbase4:39223-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. because compaction request was cancelled 2023-04-20 20:56:40,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 20:56:40,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 819071c32df35da7c00fe2675f028bf5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 20:56:40,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=A 2023-04-20 20:56:40,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:40,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=B 2023-04-20 20:56:40,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:40,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 819071c32df35da7c00fe2675f028bf5, store=C 2023-04-20 20:56:40,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 20:56:40,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304203c93dc3d358b426592880a1c23872624 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203c93dc3d358b426592880a1c23872624 2023-04-20 20:56:40,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1649, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/56f0c90eb0c648dda6567cbb487f7e3e 2023-04-20 20:56:40,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 20:56:41,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1649 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/62b613fca87f4384b52a0f9aa1b1f663 2023-04-20 20:56:41,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1649 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/ad7681e8a43740298c73d2dea22112e4 2023-04-20 20:56:41,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/A/56f0c90eb0c648dda6567cbb487f7e3e as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/56f0c90eb0c648dda6567cbb487f7e3e 2023-04-20 20:56:41,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/56f0c90eb0c648dda6567cbb487f7e3e, entries=150, sequenceid=1649, filesize=25.4 K 2023-04-20 20:56:41,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/B/62b613fca87f4384b52a0f9aa1b1f663 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/62b613fca87f4384b52a0f9aa1b1f663 2023-04-20 20:56:41,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/62b613fca87f4384b52a0f9aa1b1f663, entries=150, sequenceid=1649, filesize=11.9 K 2023-04-20 20:56:41,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/.tmp/C/ad7681e8a43740298c73d2dea22112e4 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ad7681e8a43740298c73d2dea22112e4 2023-04-20 20:56:41,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ad7681e8a43740298c73d2dea22112e4, entries=150, sequenceid=1649, filesize=11.9 K 2023-04-20 20:56:41,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 819071c32df35da7c00fe2675f028bf5 in 1241ms, sequenceid=1649, compaction requested=true 2023-04-20 20:56:41,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f80b87db520b445e891f3edb91f672f3, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/37704a6c3d6a4e37a1e05edb02d49950, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8c440ff8a713478291f693ce0a510c1a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f91fd0af652449a4a2698be348448d1a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8af9b807bbf0415c8f0943fe0515401b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c212933e32b341f881275d3fdd424064, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/cf6ca19fec014dbd9e2b0b01f1da8794, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e1af8b6aea44403bbf4b0f615707b0ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f7bf718834814c538e6e02175eab76a8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/9ee2c10afdf44993a717230dd70719ed, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/b8dae304f72043b1bfe0a8ba13a98100, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0cfeeb95d9634a4b993c0384568cfbe6, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/7c1c2e7363a045e4b267c30323749141, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6037f07d1d9e46fb8ee8a5ce9b870076, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/1683d40670104f5196fbf25657da4e3c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6fb5ba04dd34496b9798d6cebea15d62, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0b6bce97acf84c0a9d24708b59795782, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/80e5c4903d3b4270b3a292c6e20014c7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2b883f3a48ab4f12bca74f87ba8250f7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d20a7cd5588646b2a767a262a7e61b36, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/24d9438c0890458286dc421e3d150164, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fb2610d32e734357bc3b4d109d6474e8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/25cb61998af84aafa6e4d4972c01322f, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d0132fc7b67947dda24dbe2dd5cc09c1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0434f3d3dbd54b78a3aff4fadfc48535, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/44f4684fda0242ec9e61b55b1be90088, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e96f35cf969447edb2b0d3a94c42d72d, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fa9dd411c3d846bcab7acc89e7342738, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/30f3332b312043a48eaca080c6f4bdef, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2186ffcace36427da39cf6a7b6eaab84] to archive 2023-04-20 20:56:41,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:56:41,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f80b87db520b445e891f3edb91f672f3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f80b87db520b445e891f3edb91f672f3 2023-04-20 20:56:41,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/37704a6c3d6a4e37a1e05edb02d49950 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/37704a6c3d6a4e37a1e05edb02d49950 2023-04-20 20:56:41,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8c440ff8a713478291f693ce0a510c1a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8c440ff8a713478291f693ce0a510c1a 2023-04-20 20:56:41,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f91fd0af652449a4a2698be348448d1a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f91fd0af652449a4a2698be348448d1a 2023-04-20 20:56:41,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8af9b807bbf0415c8f0943fe0515401b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/8af9b807bbf0415c8f0943fe0515401b 2023-04-20 20:56:41,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c212933e32b341f881275d3fdd424064 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c212933e32b341f881275d3fdd424064 2023-04-20 20:56:41,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/cf6ca19fec014dbd9e2b0b01f1da8794 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/cf6ca19fec014dbd9e2b0b01f1da8794 2023-04-20 20:56:41,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e1af8b6aea44403bbf4b0f615707b0ad to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e1af8b6aea44403bbf4b0f615707b0ad 2023-04-20 20:56:41,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f7bf718834814c538e6e02175eab76a8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/f7bf718834814c538e6e02175eab76a8 2023-04-20 20:56:41,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/9ee2c10afdf44993a717230dd70719ed to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/9ee2c10afdf44993a717230dd70719ed 2023-04-20 20:56:41,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/b8dae304f72043b1bfe0a8ba13a98100 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/b8dae304f72043b1bfe0a8ba13a98100 2023-04-20 20:56:41,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0cfeeb95d9634a4b993c0384568cfbe6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0cfeeb95d9634a4b993c0384568cfbe6 2023-04-20 20:56:41,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/7c1c2e7363a045e4b267c30323749141 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/7c1c2e7363a045e4b267c30323749141 2023-04-20 20:56:41,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6037f07d1d9e46fb8ee8a5ce9b870076 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6037f07d1d9e46fb8ee8a5ce9b870076 2023-04-20 20:56:41,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/1683d40670104f5196fbf25657da4e3c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/1683d40670104f5196fbf25657da4e3c 2023-04-20 20:56:41,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6fb5ba04dd34496b9798d6cebea15d62 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/6fb5ba04dd34496b9798d6cebea15d62 2023-04-20 20:56:41,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0b6bce97acf84c0a9d24708b59795782 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0b6bce97acf84c0a9d24708b59795782 2023-04-20 20:56:41,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/80e5c4903d3b4270b3a292c6e20014c7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/80e5c4903d3b4270b3a292c6e20014c7 2023-04-20 20:56:41,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2b883f3a48ab4f12bca74f87ba8250f7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2b883f3a48ab4f12bca74f87ba8250f7 2023-04-20 20:56:41,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d20a7cd5588646b2a767a262a7e61b36 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d20a7cd5588646b2a767a262a7e61b36 2023-04-20 20:56:41,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/24d9438c0890458286dc421e3d150164 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/24d9438c0890458286dc421e3d150164 2023-04-20 20:56:41,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fb2610d32e734357bc3b4d109d6474e8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fb2610d32e734357bc3b4d109d6474e8 2023-04-20 20:56:41,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/25cb61998af84aafa6e4d4972c01322f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/25cb61998af84aafa6e4d4972c01322f 2023-04-20 20:56:41,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d0132fc7b67947dda24dbe2dd5cc09c1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d0132fc7b67947dda24dbe2dd5cc09c1 2023-04-20 20:56:41,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0434f3d3dbd54b78a3aff4fadfc48535 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/0434f3d3dbd54b78a3aff4fadfc48535 2023-04-20 20:56:41,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/44f4684fda0242ec9e61b55b1be90088 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/44f4684fda0242ec9e61b55b1be90088 2023-04-20 20:56:41,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e96f35cf969447edb2b0d3a94c42d72d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/e96f35cf969447edb2b0d3a94c42d72d 2023-04-20 20:56:41,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fa9dd411c3d846bcab7acc89e7342738 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/fa9dd411c3d846bcab7acc89e7342738 2023-04-20 20:56:41,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/30f3332b312043a48eaca080c6f4bdef to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/30f3332b312043a48eaca080c6f4bdef 2023-04-20 20:56:41,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2186ffcace36427da39cf6a7b6eaab84 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/2186ffcace36427da39cf6a7b6eaab84 2023-04-20 20:56:41,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3672022998644506991fab3b90063485, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2cdb3409d3d64ebba9701ce1b01d40c7, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/15c574e6f2314102b604a1a150eccb5e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f80ea50a0dbc4ffcb320db6ad965a398, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f68b9643729c419ba13c87f8c46dba89, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2991e7d490dd4b4ea6bc3fbd660c4576, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5dfc7bb73912404e806270e7be19f727, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f1099b3d23be4e9a97f940cd4645c02e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/07439551088440eda12c2e2893e2099c, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/906442be2899453e97e26258da556b71, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/17689612029f4422be45fb733c230f84, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b534188b1d8a4686b4b4fe27e737e10b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/05b32cc6b31743ef94320ad751cb21ba, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/830bc03e57244fe8bc3cb2ef162240b5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/db5328af4d6242c7b8bd5e31cbec4d44, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/4eb2780f57c845bba935398d8eeae271, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3d6ad2996c5742e8b524370df9323bbe, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/99a44a6830b640299a5fff4b79e93acb, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/dae0a27ea8794bc5b9569699e78585c9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/003ec7aea6f443659db9b8956f567866, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3c48fded5cfc4ab59040f1c0c3213aa4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b9285ed8adf744aead22e911eb60b615, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/0f5cad6420e74d86b8cf7499e2e36819, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5bee712c63354234ac33357ed08fc6cc, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ca185c6f84f84391afc130359bd03f1a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/c3f6300f13de4ff5afc173f5eb4a5c13, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2690dd63670047daa3447424ea3d587a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/23b547c45efd4ed294c8834d26111ee4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2a886d0371b14376aa0d07715c9febb9, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/d2a71b5807b8461cb389a2601667fb7b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/8a75688a5935454fa553484341decb4e] to archive 2023-04-20 20:56:41,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:56:41,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3672022998644506991fab3b90063485 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3672022998644506991fab3b90063485 2023-04-20 20:56:41,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2cdb3409d3d64ebba9701ce1b01d40c7 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2cdb3409d3d64ebba9701ce1b01d40c7 2023-04-20 20:56:41,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/15c574e6f2314102b604a1a150eccb5e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/15c574e6f2314102b604a1a150eccb5e 2023-04-20 20:56:41,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f80ea50a0dbc4ffcb320db6ad965a398 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f80ea50a0dbc4ffcb320db6ad965a398 2023-04-20 20:56:41,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f68b9643729c419ba13c87f8c46dba89 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f68b9643729c419ba13c87f8c46dba89 2023-04-20 20:56:41,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2991e7d490dd4b4ea6bc3fbd660c4576 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2991e7d490dd4b4ea6bc3fbd660c4576 2023-04-20 20:56:41,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5dfc7bb73912404e806270e7be19f727 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5dfc7bb73912404e806270e7be19f727 2023-04-20 20:56:41,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f1099b3d23be4e9a97f940cd4645c02e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/f1099b3d23be4e9a97f940cd4645c02e 2023-04-20 20:56:41,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/07439551088440eda12c2e2893e2099c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/07439551088440eda12c2e2893e2099c 2023-04-20 20:56:41,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/906442be2899453e97e26258da556b71 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/906442be2899453e97e26258da556b71 2023-04-20 20:56:41,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/17689612029f4422be45fb733c230f84 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/17689612029f4422be45fb733c230f84 2023-04-20 20:56:41,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b534188b1d8a4686b4b4fe27e737e10b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b534188b1d8a4686b4b4fe27e737e10b 2023-04-20 20:56:41,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/05b32cc6b31743ef94320ad751cb21ba to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/05b32cc6b31743ef94320ad751cb21ba 2023-04-20 20:56:41,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/830bc03e57244fe8bc3cb2ef162240b5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/830bc03e57244fe8bc3cb2ef162240b5 2023-04-20 20:56:41,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/db5328af4d6242c7b8bd5e31cbec4d44 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/db5328af4d6242c7b8bd5e31cbec4d44 2023-04-20 20:56:41,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/4eb2780f57c845bba935398d8eeae271 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/4eb2780f57c845bba935398d8eeae271 2023-04-20 20:56:41,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3d6ad2996c5742e8b524370df9323bbe to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3d6ad2996c5742e8b524370df9323bbe 2023-04-20 20:56:41,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/99a44a6830b640299a5fff4b79e93acb to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/99a44a6830b640299a5fff4b79e93acb 2023-04-20 20:56:41,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/dae0a27ea8794bc5b9569699e78585c9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/dae0a27ea8794bc5b9569699e78585c9 2023-04-20 20:56:41,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/003ec7aea6f443659db9b8956f567866 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/003ec7aea6f443659db9b8956f567866 2023-04-20 20:56:41,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3c48fded5cfc4ab59040f1c0c3213aa4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/3c48fded5cfc4ab59040f1c0c3213aa4 2023-04-20 20:56:41,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b9285ed8adf744aead22e911eb60b615 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/b9285ed8adf744aead22e911eb60b615 2023-04-20 20:56:41,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/0f5cad6420e74d86b8cf7499e2e36819 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/0f5cad6420e74d86b8cf7499e2e36819 2023-04-20 20:56:41,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5bee712c63354234ac33357ed08fc6cc to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5bee712c63354234ac33357ed08fc6cc 2023-04-20 20:56:41,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ca185c6f84f84391afc130359bd03f1a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ca185c6f84f84391afc130359bd03f1a 2023-04-20 20:56:41,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/c3f6300f13de4ff5afc173f5eb4a5c13 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/c3f6300f13de4ff5afc173f5eb4a5c13 2023-04-20 20:56:41,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2690dd63670047daa3447424ea3d587a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2690dd63670047daa3447424ea3d587a 2023-04-20 20:56:41,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/23b547c45efd4ed294c8834d26111ee4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/23b547c45efd4ed294c8834d26111ee4 2023-04-20 20:56:41,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2a886d0371b14376aa0d07715c9febb9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/2a886d0371b14376aa0d07715c9febb9 2023-04-20 20:56:41,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/d2a71b5807b8461cb389a2601667fb7b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/d2a71b5807b8461cb389a2601667fb7b 2023-04-20 20:56:41,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/8a75688a5935454fa553484341decb4e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/8a75688a5935454fa553484341decb4e 2023-04-20 20:56:41,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9d3abe677ed54ec2a9ea0e9f19794626, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3bad02064bd54d8bb4075fec95ab5832, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8197825cfa4542c8b866769249d5474e, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/13645249b80a49be8e14ba788d36f468, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/de905861e6ac43f9aab887fbcfaa6f5b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55b7cbb792554dd8bb2a3110b17a8a5a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/39fd6f6e8cb141599b7a565ff0d1b0ad, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8fc3e148d0a547a1b88ad290ac3e4428, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/7903923367ee43009a14cf8a230f0286, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/120fef011cb44229915a73de3d97a505, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e41a76bfc9564698859f170adb121e82, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1833c889288f4bdc9bde1a3874c71902, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/97882247397144f78fb0fddf47800dd0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/adb476adccf0456b8e8c0ac33a070fa5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1bb29498372e454db538969c64356969, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/95edeb826ece4ce58b4e6275744e6416, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9e96caffcccd49cc8a3fb23eaa22f98b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/cc27dabe4434407ead1de06183cca8a8, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b9948ad2555d4455923bac0f69c45ca0, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/530c19bb0a614739a37a0fd461722697, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9315636958ba4bffbf961bbe75f86db2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/30d318a5665044b7a224e9f765f9aaa4, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/76a16bf719414227941e5b5a4b1eb35a, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/66397155b68d428dbec9938190cdc27b, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ed5a6cb9a97546728136bcaea1f99537, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e0196d4ad8d645babcc8f40d1ec19dc5, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/29b7705b767a43d4b96908b69c7550c1, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/5e101497b13a46019cb0d9e27edf3bc2, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3c01463fedde4d29853ffd78587d3b27] to archive 2023-04-20 20:56:41,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 20:56:41,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9d3abe677ed54ec2a9ea0e9f19794626 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9d3abe677ed54ec2a9ea0e9f19794626 2023-04-20 20:56:41,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3bad02064bd54d8bb4075fec95ab5832 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3bad02064bd54d8bb4075fec95ab5832 2023-04-20 20:56:41,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8197825cfa4542c8b866769249d5474e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8197825cfa4542c8b866769249d5474e 2023-04-20 20:56:41,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/13645249b80a49be8e14ba788d36f468 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/13645249b80a49be8e14ba788d36f468 2023-04-20 20:56:41,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/de905861e6ac43f9aab887fbcfaa6f5b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/de905861e6ac43f9aab887fbcfaa6f5b 2023-04-20 20:56:41,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55b7cbb792554dd8bb2a3110b17a8a5a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55b7cbb792554dd8bb2a3110b17a8a5a 2023-04-20 20:56:41,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/39fd6f6e8cb141599b7a565ff0d1b0ad to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/39fd6f6e8cb141599b7a565ff0d1b0ad 2023-04-20 20:56:41,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8fc3e148d0a547a1b88ad290ac3e4428 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/8fc3e148d0a547a1b88ad290ac3e4428 2023-04-20 20:56:41,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/7903923367ee43009a14cf8a230f0286 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/7903923367ee43009a14cf8a230f0286 2023-04-20 20:56:41,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/120fef011cb44229915a73de3d97a505 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/120fef011cb44229915a73de3d97a505 2023-04-20 20:56:41,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e41a76bfc9564698859f170adb121e82 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e41a76bfc9564698859f170adb121e82 2023-04-20 20:56:41,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1833c889288f4bdc9bde1a3874c71902 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1833c889288f4bdc9bde1a3874c71902 2023-04-20 20:56:41,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/97882247397144f78fb0fddf47800dd0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/97882247397144f78fb0fddf47800dd0 2023-04-20 20:56:41,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/adb476adccf0456b8e8c0ac33a070fa5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/adb476adccf0456b8e8c0ac33a070fa5 2023-04-20 20:56:41,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1bb29498372e454db538969c64356969 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/1bb29498372e454db538969c64356969 2023-04-20 20:56:41,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/95edeb826ece4ce58b4e6275744e6416 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/95edeb826ece4ce58b4e6275744e6416 2023-04-20 20:56:41,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9e96caffcccd49cc8a3fb23eaa22f98b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9e96caffcccd49cc8a3fb23eaa22f98b 2023-04-20 20:56:41,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/cc27dabe4434407ead1de06183cca8a8 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/cc27dabe4434407ead1de06183cca8a8 2023-04-20 20:56:41,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b9948ad2555d4455923bac0f69c45ca0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b9948ad2555d4455923bac0f69c45ca0 2023-04-20 20:56:41,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/530c19bb0a614739a37a0fd461722697 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/530c19bb0a614739a37a0fd461722697 2023-04-20 20:56:41,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9315636958ba4bffbf961bbe75f86db2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/9315636958ba4bffbf961bbe75f86db2 2023-04-20 20:56:41,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/30d318a5665044b7a224e9f765f9aaa4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/30d318a5665044b7a224e9f765f9aaa4 2023-04-20 20:56:41,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/76a16bf719414227941e5b5a4b1eb35a to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/76a16bf719414227941e5b5a4b1eb35a 2023-04-20 20:56:41,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/66397155b68d428dbec9938190cdc27b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/66397155b68d428dbec9938190cdc27b 2023-04-20 20:56:41,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ed5a6cb9a97546728136bcaea1f99537 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ed5a6cb9a97546728136bcaea1f99537 2023-04-20 20:56:41,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e0196d4ad8d645babcc8f40d1ec19dc5 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/e0196d4ad8d645babcc8f40d1ec19dc5 2023-04-20 20:56:41,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/29b7705b767a43d4b96908b69c7550c1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/29b7705b767a43d4b96908b69c7550c1 2023-04-20 20:56:41,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/5e101497b13a46019cb0d9e27edf3bc2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/5e101497b13a46019cb0d9e27edf3bc2 2023-04-20 20:56:41,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3c01463fedde4d29853ffd78587d3b27 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3c01463fedde4d29853ffd78587d3b27 2023-04-20 20:56:41,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/recovered.edits/1652.seqid, newMaxSeqId=1652, maxSeqId=4 2023-04-20 20:56:41,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5. 2023-04-20 20:56:41,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 819071c32df35da7c00fe2675f028bf5: 2023-04-20 20:56:41,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:41,609 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=819071c32df35da7c00fe2675f028bf5, regionState=CLOSED 2023-04-20 20:56:41,610 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682024201609"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682024201609"}]},"ts":"1682024201609"} 2023-04-20 20:56:41,611 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-20 20:56:41,611 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 819071c32df35da7c00fe2675f028bf5, server=jenkins-hbase4.apache.org,39223,1682024020002 in 2.0120 sec 2023-04-20 20:56:41,612 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-20 20:56:41,612 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=819071c32df35da7c00fe2675f028bf5, UNASSIGN in 2.0160 sec 2023-04-20 20:56:41,613 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682024201612"}]},"ts":"1682024201612"} 2023-04-20 20:56:41,613 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 20:56:41,615 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 20:56:41,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0250 sec 2023-04-20 20:56:41,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 20:56:41,697 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-20 20:56:41,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-20 20:56:41,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:41,700 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:41,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-20 20:56:41,700 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:41,704 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:41,706 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C, FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/recovered.edits] 2023-04-20 20:56:41,708 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/56f0c90eb0c648dda6567cbb487f7e3e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/56f0c90eb0c648dda6567cbb487f7e3e 2023-04-20 20:56:41,709 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/a911c6d14a664b7985abc71f7bff57b9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/a911c6d14a664b7985abc71f7bff57b9 2023-04-20 20:56:41,710 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c44e6b5d31ce4d949378c3db925897a2 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/c44e6b5d31ce4d949378c3db925897a2 2023-04-20 20:56:41,711 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d52b7bf923f5406e8118db4912e855ae to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/A/d52b7bf923f5406e8118db4912e855ae 2023-04-20 20:56:41,713 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5355d07fa28d475fa8a4f867e8482ec1 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/5355d07fa28d475fa8a4f867e8482ec1 2023-04-20 20:56:41,714 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/55e60db9097c4e9c9fac6aaf1592a000 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/55e60db9097c4e9c9fac6aaf1592a000 2023-04-20 20:56:41,716 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/62b613fca87f4384b52a0f9aa1b1f663 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/62b613fca87f4384b52a0f9aa1b1f663 2023-04-20 20:56:41,717 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ceff4e37a0f84336a9a8d448c65ced84 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/B/ceff4e37a0f84336a9a8d448c65ced84 2023-04-20 20:56:41,719 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3eb6a263700f4453b7ba28f6aff401bd to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/3eb6a263700f4453b7ba28f6aff401bd 2023-04-20 20:56:41,720 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55e8d0a22ccc432a95ff5d6354925a68 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/55e8d0a22ccc432a95ff5d6354925a68 2023-04-20 20:56:41,721 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ab9e8b457e744c32b82e1247d997832f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ab9e8b457e744c32b82e1247d997832f 2023-04-20 20:56:41,722 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ad7681e8a43740298c73d2dea22112e4 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/ad7681e8a43740298c73d2dea22112e4 2023-04-20 20:56:41,723 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b4fb5e9bfb244fc69e99a34a10fcecf3 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/C/b4fb5e9bfb244fc69e99a34a10fcecf3 2023-04-20 20:56:41,724 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/recovered.edits/1652.seqid to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5/recovered.edits/1652.seqid 2023-04-20 20:56:41,725 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/.tmp/data/default/TestAcidGuarantees/819071c32df35da7c00fe2675f028bf5 2023-04-20 20:56:41,725 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 20:56:41,725 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 20:56:41,726 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-20 20:56:41,728 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200c4f614c94094c8e95a5f60affbe2b89 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200c4f614c94094c8e95a5f60affbe2b89 2023-04-20 20:56:41,729 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201920e3d4b31a494c95e898d640c35029 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201920e3d4b31a494c95e898d640c35029 2023-04-20 20:56:41,729 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420248aec14d26f462bae9c3788b30eb709 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420248aec14d26f462bae9c3788b30eb709 2023-04-20 20:56:41,730 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203c93dc3d358b426592880a1c23872624 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203c93dc3d358b426592880a1c23872624 2023-04-20 20:56:41,731 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420456ef784a2254031b5f95f5048f9997e to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420456ef784a2254031b5f95f5048f9997e 2023-04-20 20:56:41,732 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042045710b545616454096b7b647d89f5314 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042045710b545616454096b7b647d89f5314 2023-04-20 20:56:41,733 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304204945030d552e4ea894e6728bf898a8ca to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304204945030d552e4ea894e6728bf898a8ca 2023-04-20 20:56:41,733 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420685b00de68214d88942db59ff8f3f273 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420685b00de68214d88942db59ff8f3f273 2023-04-20 20:56:41,734 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207f03b0b8dc294354834e311425561b32 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207f03b0b8dc294354834e311425561b32 2023-04-20 20:56:41,735 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042092564d2e04a741b2858acb99e6903e30 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042092564d2e04a741b2858acb99e6903e30 2023-04-20 20:56:41,736 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209574d7c513524f9eb2ea34bd9dc00a67 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209574d7c513524f9eb2ea34bd9dc00a67 2023-04-20 20:56:41,737 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042096a96c35b08e4e208e10a06265e64951 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042096a96c35b08e4e208e10a06265e64951 2023-04-20 20:56:41,738 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042096e23ccb6e024077931422eeb607696c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042096e23ccb6e024077931422eeb607696c 2023-04-20 20:56:41,739 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ab7e2b57ca1342c8be1264c1b1b2c51b to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ab7e2b57ca1342c8be1264c1b1b2c51b 2023-04-20 20:56:41,740 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b14ab756707244ddab48b8bfb8a48ef6 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b14ab756707244ddab48b8bfb8a48ef6 2023-04-20 20:56:41,740 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b1e8074a57484d059c376aa11edf5fe0 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b1e8074a57484d059c376aa11edf5fe0 2023-04-20 20:56:41,741 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b64baa1a14e542ed82afc9f86e3eb94c to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b64baa1a14e542ed82afc9f86e3eb94c 2023-04-20 20:56:41,742 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ba78a63224ca4105a99673ea47110ab9 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ba78a63224ca4105a99673ea47110ab9 2023-04-20 20:56:41,742 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c331866ff22f44aaa20b29d502d49400 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c331866ff22f44aaa20b29d502d49400 2023-04-20 20:56:41,743 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c52c18a5bc794e6699de1181a46ca478 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c52c18a5bc794e6699de1181a46ca478 2023-04-20 20:56:41,744 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c5780cfc30d340d082efb9f30c436a66 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c5780cfc30d340d082efb9f30c436a66 2023-04-20 20:56:41,745 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ce6599e61b4e4dc095eec60035e3310f to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ce6599e61b4e4dc095eec60035e3310f 2023-04-20 20:56:41,746 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d007738f9aa04728ace206a43883793d to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d007738f9aa04728ace206a43883793d 2023-04-20 20:56:41,746 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d05a85aa380d4576800efe021de93b65 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d05a85aa380d4576800efe021de93b65 2023-04-20 20:56:41,747 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d145101028e54256849e5e0250bfb1cf to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d145101028e54256849e5e0250bfb1cf 2023-04-20 20:56:41,748 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d9ee614783d547e4afc1d0b4802bc0ea to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d9ee614783d547e4afc1d0b4802bc0ea 2023-04-20 20:56:41,749 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420edfdf7c1b57742f281b8183c88d89773 to hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420edfdf7c1b57742f281b8183c88d89773 2023-04-20 20:56:41,749 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 20:56:41,751 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:41,752 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 20:56:41,754 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 20:56:41,754 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:41,754 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 20:56:41,754 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682024201754"}]},"ts":"9223372036854775807"} 2023-04-20 20:56:41,756 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 20:56:41,756 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 819071c32df35da7c00fe2675f028bf5, NAME => 'TestAcidGuarantees,,1682024173484.819071c32df35da7c00fe2675f028bf5.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 20:56:41,756 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 20:56:41,756 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682024201756"}]},"ts":"9223372036854775807"} 2023-04-20 20:56:41,757 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 20:56:41,759 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 20:56:41,759 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 61 msec 2023-04-20 20:56:41,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44655] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-20 20:56:41,801 INFO [Listener at localhost/45619] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-20 20:56:41,804 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-20 20:56:41,813 INFO [Listener at localhost/45619] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 228) - Thread LEAK? -, OpenFileDescriptor=580 (was 609), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=399 (was 409), ProcessCount=169 (was 169), AvailableMemoryMB=9092 (was 9060) - AvailableMemoryMB LEAK? - 2023-04-20 20:56:41,813 INFO [Listener at localhost/45619] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-20 20:56:41,813 INFO [Listener at localhost/45619] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 20:56:41,814 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:62055 2023-04-20 20:56:41,814 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:41,814 DEBUG [Listener at localhost/45619] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-20 20:56:41,814 DEBUG [Listener at localhost/45619] util.JVMClusterUtil(257): Found active master hash=1075419909, stopped=false 2023-04-20 20:56:41,814 INFO [Listener at localhost/45619] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:56:41,815 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-20 20:56:41,816 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-20 20:56:41,816 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:56:41,816 INFO [Listener at localhost/45619] procedure2.ProcedureExecutor(629): Stopping 2023-04-20 20:56:41,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 20:56:41,816 DEBUG [Listener at localhost/45619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fd3a1b4 to 127.0.0.1:62055 2023-04-20 20:56:41,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 20:56:41,816 DEBUG [Listener at localhost/45619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:41,817 INFO [Listener at localhost/45619] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39223,1682024020002' ***** 2023-04-20 20:56:41,817 INFO [Listener at localhost/45619] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-20 20:56:41,817 INFO [RS:0;jenkins-hbase4:39223] regionserver.HeapMemoryManager(220): Stopping 2023-04-20 20:56:41,817 INFO [RS:0;jenkins-hbase4:39223] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-20 20:56:41,817 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-20 20:56:41,817 INFO [RS:0;jenkins-hbase4:39223] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-20 20:56:41,817 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(3304): Received CLOSE for 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:56:41,818 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:41,818 DEBUG [RS:0;jenkins-hbase4:39223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7efe842c to 127.0.0.1:62055 2023-04-20 20:56:41,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3d827b8c3467151cc7f52ff57fde3810, disabling compactions & flushes 2023-04-20 20:56:41,819 DEBUG [RS:0;jenkins-hbase4:39223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:41,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:56:41,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:56:41,819 INFO [RS:0;jenkins-hbase4:39223] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-20 20:56:41,819 INFO [RS:0;jenkins-hbase4:39223] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-20 20:56:41,819 INFO [RS:0;jenkins-hbase4:39223] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-20 20:56:41,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. after waiting 0 ms 2023-04-20 20:56:41,819 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-20 20:56:41,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:56:41,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3d827b8c3467151cc7f52ff57fde3810 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-20 20:56:41,819 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-20 20:56:41,819 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 3d827b8c3467151cc7f52ff57fde3810=hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810.} 2023-04-20 20:56:41,820 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-20 20:56:41,820 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-20 20:56:41,820 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-20 20:56:41,820 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-20 20:56:41,820 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-20 20:56:41,820 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-20 20:56:41,823 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1505): Waiting on 1588230740, 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:56:41,831 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/.tmp/info/f8992235080d42a9b3556fe146c84610 2023-04-20 20:56:41,835 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f8992235080d42a9b3556fe146c84610 2023-04-20 20:56:41,859 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-20 20:56:42,023 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1505): Waiting on 1588230740, 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:56:42,224 DEBUG [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1505): Waiting on 1588230740, 3d827b8c3467151cc7f52ff57fde3810 2023-04-20 20:56:42,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/.tmp/info/b977322de7eb4d5683a1743b1d6e2a01 2023-04-20 20:56:42,236 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/.tmp/info/b977322de7eb4d5683a1743b1d6e2a01 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/info/b977322de7eb4d5683a1743b1d6e2a01 2023-04-20 20:56:42,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/info/b977322de7eb4d5683a1743b1d6e2a01, entries=2, sequenceid=6, filesize=4.8 K 2023-04-20 20:56:42,240 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/.tmp/rep_barrier/c62e0fc99dbd4898b5f8cee73934dff5 2023-04-20 20:56:42,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 3d827b8c3467151cc7f52ff57fde3810 in 421ms, sequenceid=6, compaction requested=false 2023-04-20 20:56:42,244 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c62e0fc99dbd4898b5f8cee73934dff5 2023-04-20 20:56:42,244 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/namespace/3d827b8c3467151cc7f52ff57fde3810/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-20 20:56:42,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:56:42,245 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3d827b8c3467151cc7f52ff57fde3810: 2023-04-20 20:56:42,245 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682024022744.3d827b8c3467151cc7f52ff57fde3810. 2023-04-20 20:56:42,250 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/.tmp/table/27b438e4a86b4aaa82d440b52bab1de1 2023-04-20 20:56:42,259 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 27b438e4a86b4aaa82d440b52bab1de1 2023-04-20 20:56:42,260 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/.tmp/info/f8992235080d42a9b3556fe146c84610 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/info/f8992235080d42a9b3556fe146c84610 2023-04-20 20:56:42,262 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f8992235080d42a9b3556fe146c84610 2023-04-20 20:56:42,262 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/info/f8992235080d42a9b3556fe146c84610, entries=22, sequenceid=93, filesize=7.1 K 2023-04-20 20:56:42,264 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/.tmp/rep_barrier/c62e0fc99dbd4898b5f8cee73934dff5 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/rep_barrier/c62e0fc99dbd4898b5f8cee73934dff5 2023-04-20 20:56:42,267 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c62e0fc99dbd4898b5f8cee73934dff5 2023-04-20 20:56:42,267 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/rep_barrier/c62e0fc99dbd4898b5f8cee73934dff5, entries=6, sequenceid=93, filesize=5.5 K 2023-04-20 20:56:42,267 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/.tmp/table/27b438e4a86b4aaa82d440b52bab1de1 as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/table/27b438e4a86b4aaa82d440b52bab1de1 2023-04-20 20:56:42,270 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 27b438e4a86b4aaa82d440b52bab1de1 2023-04-20 20:56:42,270 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/table/27b438e4a86b4aaa82d440b52bab1de1, entries=9, sequenceid=93, filesize=5.5 K 2023-04-20 20:56:42,270 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 450ms, sequenceid=93, compaction requested=false 2023-04-20 20:56:42,274 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-20 20:56:42,275 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-20 20:56:42,276 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-20 20:56:42,276 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-20 20:56:42,276 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-20 20:56:42,424 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39223,1682024020002; all regions closed. 2023-04-20 20:56:42,432 DEBUG [RS:0;jenkins-hbase4:39223] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/oldWALs 2023-04-20 20:56:42,432 INFO [RS:0;jenkins-hbase4:39223] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39223%2C1682024020002.meta:.meta(num 1682024022458) 2023-04-20 20:56:42,436 DEBUG [RS:0;jenkins-hbase4:39223] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/oldWALs 2023-04-20 20:56:42,436 INFO [RS:0;jenkins-hbase4:39223] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39223%2C1682024020002:(num 1682024021920) 2023-04-20 20:56:42,436 DEBUG [RS:0;jenkins-hbase4:39223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:42,436 INFO [RS:0;jenkins-hbase4:39223] regionserver.LeaseManager(133): Closed leases 2023-04-20 20:56:42,436 INFO [RS:0;jenkins-hbase4:39223] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-20 20:56:42,437 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-20 20:56:42,438 INFO [RS:0;jenkins-hbase4:39223] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39223 2023-04-20 20:56:42,444 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-20 20:56:42,444 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39223,1682024020002 2023-04-20 20:56:42,444 ERROR [Listener at localhost/45619-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7229efa0 rejected from java.util.concurrent.ThreadPoolExecutor@7616753d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2697] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 20:56:42,444 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-20 20:56:42,444 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39223,1682024020002] 2023-04-20 20:56:42,444 ERROR [Listener at localhost/45619-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f0499c2 rejected from java.util.concurrent.ThreadPoolExecutor@7616753d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2697] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 20:56:42,445 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39223,1682024020002; numProcessing=1 2023-04-20 20:56:42,446 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39223,1682024020002 already deleted, retry=false 2023-04-20 20:56:42,446 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39223,1682024020002 expired; onlineServers=0 2023-04-20 20:56:42,446 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44655,1682024018782' ***** 2023-04-20 20:56:42,446 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-20 20:56:42,447 DEBUG [M:0;jenkins-hbase4:44655] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@124913e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-20 20:56:42,447 INFO [M:0;jenkins-hbase4:44655] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44655,1682024018782 2023-04-20 20:56:42,447 INFO [M:0;jenkins-hbase4:44655] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44655,1682024018782; all regions closed. 2023-04-20 20:56:42,447 DEBUG [M:0;jenkins-hbase4:44655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 20:56:42,447 DEBUG [M:0;jenkins-hbase4:44655] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-20 20:56:42,447 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-20 20:56:42,447 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682024021577] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682024021577,5,FailOnTimeoutGroup] 2023-04-20 20:56:42,447 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682024021584] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682024021584,5,FailOnTimeoutGroup] 2023-04-20 20:56:42,447 DEBUG [M:0;jenkins-hbase4:44655] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-20 20:56:42,447 INFO [M:0;jenkins-hbase4:44655] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-20 20:56:42,447 INFO [M:0;jenkins-hbase4:44655] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-20 20:56:42,447 INFO [M:0;jenkins-hbase4:44655] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-20 20:56:42,448 DEBUG [M:0;jenkins-hbase4:44655] master.HMaster(1512): Stopping service threads 2023-04-20 20:56:42,448 INFO [M:0;jenkins-hbase4:44655] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-20 20:56:42,448 ERROR [M:0;jenkins-hbase4:44655] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-20 20:56:42,449 INFO [M:0;jenkins-hbase4:44655] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-20 20:56:42,449 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-20 20:56:42,449 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-20 20:56:42,450 DEBUG [M:0;jenkins-hbase4:44655] zookeeper.ZKUtil(398): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-20 20:56:42,450 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 20:56:42,450 WARN [M:0;jenkins-hbase4:44655] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-20 20:56:42,450 INFO [M:0;jenkins-hbase4:44655] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-20 20:56:42,450 INFO [M:0;jenkins-hbase4:44655] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-20 20:56:42,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-20 20:56:42,451 DEBUG [M:0;jenkins-hbase4:44655] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-20 20:56:42,451 INFO [M:0;jenkins-hbase4:44655] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:56:42,451 DEBUG [M:0;jenkins-hbase4:44655] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:56:42,451 DEBUG [M:0;jenkins-hbase4:44655] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-20 20:56:42,451 DEBUG [M:0;jenkins-hbase4:44655] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:56:42,451 INFO [M:0;jenkins-hbase4:44655] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.10 KB heapSize=393.84 KB 2023-04-20 20:56:42,546 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 20:56:42,546 INFO [RS:0;jenkins-hbase4:39223] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39223,1682024020002; zookeeper connection closed. 2023-04-20 20:56:42,546 ERROR [Listener at localhost/45619-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@47171790 rejected from java.util.concurrent.ThreadPoolExecutor@7616753d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2697] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 20:56:42,547 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): regionserver:39223-0x101a35999ca0001, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 20:56:42,547 ERROR [Listener at localhost/45619-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@486a53f8 rejected from java.util.concurrent.ThreadPoolExecutor@7616753d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2697] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 20:56:42,547 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@279e85e5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@279e85e5 2023-04-20 20:56:42,547 INFO [Listener at localhost/45619] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-20 20:56:42,862 INFO [M:0;jenkins-hbase4:44655] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.10 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e8674cd3c1094334b80ca04b331ed2ef 2023-04-20 20:56:42,865 INFO [M:0;jenkins-hbase4:44655] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e8674cd3c1094334b80ca04b331ed2ef 2023-04-20 20:56:42,866 DEBUG [M:0;jenkins-hbase4:44655] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e8674cd3c1094334b80ca04b331ed2ef as hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e8674cd3c1094334b80ca04b331ed2ef 2023-04-20 20:56:42,868 INFO [M:0;jenkins-hbase4:44655] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e8674cd3c1094334b80ca04b331ed2ef 2023-04-20 20:56:42,868 INFO [M:0;jenkins-hbase4:44655] regionserver.HStore(1080): Added hdfs://localhost:34685/user/jenkins/test-data/0651202b-184e-d56a-566f-ecd2b1770dc9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e8674cd3c1094334b80ca04b331ed2ef, entries=65, sequenceid=579, filesize=24.7 K 2023-04-20 20:56:42,869 INFO [M:0;jenkins-hbase4:44655] regionserver.HRegion(2948): Finished flush of dataSize ~345.10 KB/353384, heapSize ~393.82 KB/403272, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 418ms, sequenceid=579, compaction requested=false 2023-04-20 20:56:42,869 INFO [M:0;jenkins-hbase4:44655] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 20:56:42,869 DEBUG [M:0;jenkins-hbase4:44655] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-20 20:56:42,872 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-20 20:56:42,872 INFO [M:0;jenkins-hbase4:44655] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-20 20:56:42,873 INFO [M:0;jenkins-hbase4:44655] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44655 2023-04-20 20:56:42,877 DEBUG [M:0;jenkins-hbase4:44655] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,44655,1682024018782 already deleted, retry=false 2023-04-20 20:56:42,978 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 20:56:42,978 INFO [M:0;jenkins-hbase4:44655] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44655,1682024018782; zookeeper connection closed. 2023-04-20 20:56:42,979 ERROR [Listener at localhost/45619-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@78bd19f5 rejected from java.util.concurrent.ThreadPoolExecutor@1498999d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 20:56:42,979 DEBUG [Listener at localhost/45619-EventThread] zookeeper.ZKWatcher(600): master:44655-0x101a35999ca0000, quorum=127.0.0.1:62055, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 20:56:42,979 ERROR [Listener at localhost/45619-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ffa6129 rejected from java.util.concurrent.ThreadPoolExecutor@1498999d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 20:56:42,981 WARN [Listener at localhost/45619] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-20 20:56:42,984 INFO [Listener at localhost/45619] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-20 20:56:43,091 WARN [BP-1172400674-172.31.14.131-1682024015396 heartbeating to localhost/127.0.0.1:34685] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-20 20:56:43,091 WARN [BP-1172400674-172.31.14.131-1682024015396 heartbeating to localhost/127.0.0.1:34685] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1172400674-172.31.14.131-1682024015396 (Datanode Uuid bab51f44-7c0d-4a22-baba-7df7faaf57e6) service to localhost/127.0.0.1:34685 2023-04-20 20:56:43,095 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/cluster_952dfd84-3d7a-970e-16de-5e4c99b46c34/dfs/data/data1/current/BP-1172400674-172.31.14.131-1682024015396] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-20 20:56:43,095 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e7906826-7b40-5149-ef9c-2e3b08097db9/cluster_952dfd84-3d7a-970e-16de-5e4c99b46c34/dfs/data/data2/current/BP-1172400674-172.31.14.131-1682024015396] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-20 20:56:43,129 INFO [Listener at localhost/45619] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-20 20:56:43,296 INFO [Listener at localhost/45619] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-20 20:56:43,469 INFO [Listener at localhost/45619] hbase.HBaseTestingUtility(1293): Minicluster is down